Topic on User talk:SamanthaNguyen

Hey, Samantha,

I'm wondering if you could talk about your StructuredNavigation extension during the next MediaWiki Stakeholder's meeting in October?

I first noticed your extension after you requested a repository shortly after i did. It looks intriguing.

When I asked what people thought about your extension in chat, River said "that seems like a huge improvement over navboxes, wow"

We typically meet online and discuss the MediaWiki happenings for the past month (example). It would be great to have you come and talk about your extension and how you've used it.

Hi Mark,

I'm glad that you all are interested in the extension, and I would love the opportunity to speak at your next meeting! Is there anything in particular you would want me to mention and discuss?

Thank you!

I am interested in how you see it being used and interested in seeing use cases in action.

River said she would like to be able to populate it from Cargo or SMW queries. And I just thought of using Lua. Are these possible? Would you be interested in adding them if they aren't right now?

Since we need to make sure we schedule enough time for you and the monthly mw news ususally takes 20-30 minutes, I'm thinking we should allocate 20-30 for you to talk about it and then, also, field questions. Does that seem right?

I don't know of any wikis that are currently using it, but I can set up a demo wiki with some use cases. Was that along the lines of what you were thinking?

I'm also more than willing to add an integration with Scribunto and Lua. Are you interested in being able to read data from a navigation, create a navigation, or both?

Unfortunately since I'm currently a university student, I may not be able to add a proper integration with Semantic MediaWiki and Cargo this month, but I would like to look into it in October.

In regards to the time allocation and questions, that sounds right, thank you.

Yes, if you could set up a demo wiki, that would be awesome. If you need some help with that, let me know.

I'll try to get River to comment here since she has more concrete ideas.

I don't think I'd ever query a navigation, rather I'd want to generate a nav by querying a Cargo table, e.g. rows of the navbox are regions, and each row is a list of tournaments in 2020 in that region.

Querying navigations seems potentially useful for generating metadata reports, but I wouldn't want to have that be part of my wiki's structure, that sounds like making the design too rigid, I'd rather have data stored in a neutral place that both the nav and whatever else needs it are querying.

@SamanthaNguyen am I right in interpreting what you said to mean that you could get Lua/Scribunto integration for building navigation done before the October meeting? If so, that would be awesome.

I just realized I may be over eager in my interpretation since you may not see the same distinction between Lua and SMW/Cargo that I see. Sorry 'bout that. Don't mean to be pushy.

@RheingoldRiver That sounds like a good and reasonable use case, thank you for explaining! Would you like it so that the data from the Cargo/SMW query is transformed into JSON, and have the extension use that query to automatically update whichever page in the Navigation namespace? This edit can be indicated as semi-automated with a registered change tag and automatic edit summary.

@MarkAHershberger So for example, Module:ExtensionJson has a giant Lua table centralizing data (auto-updated by a bot every 2 hours or so), and it auto-populates some of the data in Template:Extension (like hooks used/provided/software license etc.) with Module:Extension as the bridge.

I'm thinking that with a Lua integration, you'd be able to write a Lua table in a module page, and invoke that Lua module to render a navigation, the same way you would use a registered MW-parser tag to render a navigation. Is this correct?

I hope this clears the distinction, and that I have both uses cases down correctly.

For my part, the Lua table you're talking about makes sense. I think Cargo/SMW can also create a Lua table like that (page forms auto edit?), but I'm not sure. I'll defer to @RheingoldRiver on that bit.

@SamanthaNguyen I don't understand, how would the jsons get edited? It would be scheduled at regular intervals based on the result of the query?

@RheingoldRiver I'll be honest I'm not super familiar with SMW/Cargo, but from what I'm aware, the results are exportable. The query will be cached and there will be a refresh rate, and using MediaWiki's JobQueue functionality, they will be scheduled to update. The software will automatically handle transforming the data into a navigation.

I should clarify that the method for implementing this SMW/Cargo integration isn't set in stone, but I am willing to work on that once I have more time, and have it in a way that makes it easy to use for wiki editors. Your advice and feedback is most welcome!

Yeah that sounds like it would work. It's a bit different from what I'm used to, where there's no layer of caching the query result, the jobqueue just updates results dynamically as it refreshes pages, but I think for navigation it makes sense, since navbox content shouldn't be that dynamic. It would be good, though, if the query refresh rate can be set per navbox. Would these be pages on the wiki, i.e. could they be manually refreshed by users / an api action?

Hi @MarkAHershberger and @RheingoldRiver:

I'm going to be publishing a tentative schedule on the work for StructuredNavigation, so we're all on the same page. I would love to discuss more on integrating this in Semantic MediaWiki and Cargo during the talk, since that's when I'll have more time to focus on that feature!

I would love to be able to have at least week numbers for when I'll have certain things in place, but because I am also a university student, I'll be putting these at the very most in months. I hope you all can understand, I don't want to rush this software; I want to make sure it's high-quality, stable, and usable for everyone. I'd like to have the public demo setup available ASAP, but there are a few things that need to be done first to make it more usable.

However, If you are both fine with there being a few bugs, I can make the task for having a public wiki demo available a task higher up the list. Hopefully this clears things up.

I'm not sure if either of you have visited the Extension:StructuredNavigation page, but there is a pitfall listed (that is not on this schedule), which is that since this is in JSON and not wiki markup, the links are not considered in the pagelinks. I still need to improve on my SQL, so unfortunately this task is not listed yet in the schedule, however it is definitely on my mind.

Sept 11th (now) to end of September

1. Fixing API modules to be more compliant:
   • catch exceptions and send properly localized messages for specific error states
   • send proper HTTP headers and HTTP status codes
2. Fixing and improving JSON schema (there's some internal and external inconsistencies in the extension between the terms "page", "title", and "label"), make it more scalable
3. Adding redirect support for navigations: This will require some work since it extends the JSON content model. An example with redirect support:
   • Navigation:Dontnod is created as a redirect to Navigation:Dontnod Entertainment
   • Users are able to write <mw-navigation title="Dontnod" />, and the navigation will still render
   • Auto-generated category added to pages that use navigations that use redirects
4. Proper JSON schema validation while editing, with localized error messages
5. Adding Scribunto/Lua support: The reason this is sort of closer to last is that while it's relatively easy to add support for Scribunto, is that the things listed are before are (mostly) blocking tasks.
6. Make the demo wiki publicly available

October to November

Most of this month will probably be most likely working on the integration between Semantic MediaWiki and Cargo, since these are both huge extensions, and both of them also have a very different software architecture between each other. Semantic MediaWiki will probably be more plausible, but Cargo may not be. I will need to look into their APIs and hooks. This will be quite a huge undertaking. We can discuss the use cases and as well as performance considerations during this time.

Edit: There's actually quite a bit few more things I need to consider where to put in the schedule and figure out how to prioritize, which are:

• Visual editing interface for editors who are not familiar with JSON
• More accessible diffs based on the JSON edits

I'm currently working on a quick screen recording of the extension in the mean time, hopefully it helps!

A screen recording would be great. It gives people a quick way to learn what your extension can do before they decide to jump into reading the documentation.

I like your schedule for September. If you can really get that done in time for the user group meeting, you presentation of the extension will be really strong. That you have a plan for further development also helps.

BTW, I'm not sure if you saw this but you may want to use the SMWCon in November as a place to meet other developers and learn more about SMW.

Yes, I hope so! Here is the screen recording, a little less than a minute long (46 seconds): https://imgur.com/a/vtZfjXI

Thank you for the link. There was a typo in the URL but I just visited the SMWCon 2020 page. Right now I can't say I can attend for sure since it's a bit in the future, but I will be certainly thinking about it.

Thanks for the demo. It gives me a better idea of what you're doing. I think if you can come up with a way for lua or SMW to generate those it would be awesome.

@RheingoldRiver: does the demo change your thinking on this any?

That was pretty much how I was expecting it to work, but it's good to see it in person and have confirmation!

I've been able to do some work this week (most of it maintenance, but there are a few major patches in there): https://gerrit.wikimedia.org/r/q/project:mediawiki/extensions/StructuredNavigation+after:2020-09-21+owner:sam.t.nguyenn%2540gmail.com

API modules

To speed this up, I'm planning to have the Action API and REST API modules hidden by a feature flag, probably named like $wgStructuredNavigationEnableExperimentalAPI (or something like that), with the default value set to false.

Developers and wiki administrators who are interested can set this to true, and should acknowledge that both API modules are currently experimental/unstable (this is marked in the Action API modules, where they are marked as "internal" (which will list them as unstable in the UI when a user accesses the API endpoint, api.php). The REST API modules are also versioned at v0, and will then be moved to v1 when they are stable. When both become stable, the configuration setting will be removed and they will be enabled by default.

Copy embed feature

Currently the copy embed feature requires some JS, so I wrote a quick patch to try to make this no longer require JS, but uses progressive enhancement style to enhance the UX, by adding a clickable button to allow copying the text to the clipboard. More details are listed here: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/StructuredNavigation/+/629248

MW version

This may require 1.35 at some point, but I haven't fully decided the compatibility policy yet. As of now, this just requires 1.34.

JSON schema

I'm thinking of changing the schema to fix the consistencies, but this may make it more tedious to edit and create navigations, so I'm currently sitting on it. I might skip this for now.

Lua/Scribunto

Still working on this!

1.35 also has Parsoid PHP shipped by default, and I've already spoken with a member of the Parsoid team about how to move the code related to the extension tag to use Parsoid's API (replacing MW's PHP core Parser and related classes like PPFrame, ParserOutput, etc).

Migrating to Parsoid is something I'm looking down the road, but it doesn't have a place on the schedule just yet.

Sorry for all the notifications (should be the last thing): For regards to the wiki setup, I'm looking at having an instance up using Wikimedia Cloud soon.

@MarkAHershberger @RheingoldRiver: There is currently an experimental patch on Gerrit now for adding support for Scribunto + Lua: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/StructuredNavigation/+/629498

When the patch is ready, I'll send another screen recording showcasing a demo of it.

@SamanthaNguyen, I didn't log in for a few days and missed these notifications, but I'm really happy to see that this is progressing!

Heads up that the meeting is this Friday.