Theme: Architecting our code for change and sustainability

Type: Technical Challenges

Leader(s): Raz Shuty

Facilitator: John Bennett

Scribe: Irene

Description: As MediaWiki moves towards an API-first architecture, we need to maintain compatibility for legacy clients as well as extensions that still produce HTML directly. In this session, we’ll discuss when, where and how data gets turned into HTML, and what compatibility layers are needed for this. - phab:T206070.

Questions to answer during this session

Question	Significance: Why is this question important? What is blocked by it remaining unanswered?
When, where, and how is data turned in to HTML in MediaWiki (other than in the parser itself)?	The wikitext parser is the primary way that data gets turned in to HTML within the context of MediaWiki. SpecialPages and Action handlers are other ways. Enumerating the paths that result in rendered content is a necessary step towards an API-first world.
While we move towards an API-first architecture, how do we serve consumers that do not support client-side rendering?	For the foreseeable future, we will need to support clients that do not support client-side rendering. To do so, we need to decide where the rendering can happen for such clients, whether it can use the same code/templates as the client side rendering, how it is cached, etc.
While we move towards an API-first architecture, how do we handle extensions (and core code) that cannot be quickly ported to serve JSON instead of rendered HTML?	Changing application logic from emitting HTML to emitting data structures is a gradual process. Client-side rendering mechanisms will need a mechanism to integrate HTML generated by legacy code.
When implementing client-side rendering, how do we minimize callbacks to the server side API for miscellaneous tasks like rendering localization messages?	One major roadblock for client-side logic has been the fact that MediaWiki’s localization mechanism is based on WikiText, and needs a WikiText parser to function. Listing alternative and options for this issue is essential to planning the migration to an API-first architecture.

Questions and answers

Please write in your original questions. If you came up with additional important questions that you answered, please also write them in. (Do not include “new” questions that you did not answer, instead add them to the new questions section)

Q: When, where, and how is data turned in to HTML in MediaWiki (other than in the parser itself)?
A (If you were unable to answer the question, why not?): Skin, Action handlers, Special pages

Q: While we move towards an API-first architecture, how do we serve consumers that do not support client side rendering?
A (If you were unable to answer the question, why not?): Rendering on the server side, perhaps using the same kind of templating that is used on the client. Unclear whether “server side” mans in PHP or somewhere else.

Q: While we move towards an API-first architecture, how do we handle extensions (and core code) that cannot be quickly ported to serve JSON instead of rendered HTML?
A (If you were unable to answer the question, why not?): They will continue to render HTML. Whether this is made available as a full page via index.php, or in a more granular form somehow, is unclear.

Q: When implementing client-side rendering, how do we minimize callbacks to the server side API for miscellaneous tasks like rendering localization messages?
A (If you were unable to answer the question, why not?): Move away from wikitext for localization messages, or define a subset that can easily be rendered on the client.

Features and goals

For Use Case and Product Sessions: Given your discussion of the topic and answering questions for this session, list one or more user stories or user facing features that we should strive to deliver
1. Fully separate the presentation layer from the application logic in MediaWIki core
Why should we do this? To make it easy to build specialized UIs for different use cases. There was no full agreement on this. There was some scepticism about “API first”. Why should we do that? Who needs an alternative UI e.g. for blocking users?	What is blocking it? Lots and lots of code that directly generates HTML directly from DB queries. In core and in extensions.	Who is responsible? unclear.
2. Provide some mechanism for serving rendered bits of HTML to the client
Why should we do this? Because we will not be able to port all code to be fully API based right away. There was no full agreement on whether this is needed. If page assembly always happens in PHP, there is no (big) need for this.	What is blocking it? A specification for an entry point that allows this. Decisions on the necessary granularity. Decision on scope (skin, content, rows in listings, etc). Definition of the relationship with mobile content. Defining a caching strategy.	Who is responsible? Unclear. Needs input from various teams.
3. Have a templating (and i18n) mechanisms that can be used server side and client side.
Why should we do this? To provide consistent rendering, and be able to dynamically update a page on the client that was initially loaded in full from the server.	What is blocking it? Agreement on a templating mechanism. Availability of an i18n mechanism.	Who is responsible? unclear.

Important decisions to make

What are the most important decisions that need to be made regarding this topic?
1. How important is it to enable curation (history, patrol, diff, revert, protect, block) on the client?
Why is this important? If this is important, this drives the separation of the presentation layer. If it is not, it’s unclear if the effort is justified.	What is it blocking? It’s not even clear who owns the user experience of such activities.	Who is responsible? unclear.
2. How important is it to enforce parity between special pages and APIs (data first)?
Why is this important? If this is important, this drives the separation of the presentation layer. If it is not, it’s unclear if the effort is justified.	What is it blocking? It’s not clear who owns the API as a product. If we do data as a service, who owns the service?	Who is responsible? unclear.
3. How important is it to provide a unified experience across devices?
Why is this important? If this is important, this drives the separation of the presentation layer. If it is not, it’s unclear if the effort is justified.	What is it blocking? Product owners of mobile and desktop experience agreeing on cost and benefits for the user. Engineers who are maintaining the respective code and services agreeing on cost and benefits for themselves.	Who is responsible?

Detailed notes

Brion: What are the problems this is trying to solve, and how does [?] differ from API-first architecture?

Daniel: The problem is, replace what mobilefrontend is doing, and refine what mobile content services is doing. If we go api first, we send data to the client and render it there, but we still want to do that on the server, but do we reimplement that or use the same code. This is not about rendering wikitext. Thinking about how to split it up, we omit data from api, we want to show html; how? (It would have been helpful if I had written this up before the conference: gdoc: Frontend Architecture Needs and Questions )

GROUP 1 - Participants (optional): Daniel, Nick, Florian, Moritz, ? , Tim

Discussion:

Let's start with Ideal world, and adopt it to what we have.
In an ideal world, what bits and pieces do we have?
[?] Serve as a different purpose?
Start at top-levels
[?] what are constituents, special pages generate html and metadata. Where and how do we render article history, contribs, the form for protecting a page.
M: That's another data pipe. What are the elements we want to spread.
D: There are data pipes that can easy be rendered on the client, versus server. We could split it up and serve ToC and categories and lead sentence separately, but how to render the html for all the metadata, forms interactions UI elements
D: We will have to end up with tradeoffs,
F: Currently we render html serverside - is that what we want to do in idealworld?
We could have internal API call to render
?: Can we separate the metadata like pageprotection blocking data. Things that only get information,
D: So, separate API for reading/consumption?
?: Not necessarily, but
D: Article history and etc could use [graphql?], could also use that to modify data but maybe not best choice.
D: Where would they be turned into HTML/pixels?
F: As close to user as possible.
T: A bunch of endpoints in MW that assembles it as close to suer? But we can't do that, otherwise sending multiple requests across the network. There are trade offs for moving things closer to server.
DECISION: We need to find the optimally close point to the end-user. Determine
R: is it putting something in maintenance mode and then re-architecting?
M: Do we consider external sources as well? E.g, article ahs metadata with citations, but go to google scholar for citations…
D: That's possible. Multiple possible client applications, and some of them could do that.
T: What is the adva of assembling API requests outside of MW>
M: Access to more knowledge. DOn't have to put everything into the system. Have some value of trust.
D: Advantages are serve the same content to everyone and 'they' decide what to show.
T: We can't serve a whole lot of chunks to the client. There's no client that needs that. Except maybe for… end tail? But not mobile app, mobilefrontend, or desktop.
D: If we want to enable people to build their own interface [...]
F: Mobile apps can't show history page because there's no easy way to get the data without the html
T: How should they fix that? They could make multiple api requests (inefficiently) or could have special api endpoint (structured form), or could have bundled [...?]. But the cost of it is having 2 different systems we have to support.
D: So do you think we should stick to the desktop model of serving html to the client.
T: we should serve structured data to the client. An extension in MW for the mobile app. The team [ codes] to get
D: Maintaining 2 UXs, mobile devices and desktop. That seems like a lot of duplicate effort.
T: Hypothetically, we don't have to have a mobile app,
D: [?] don't always have to reload everything, just reload the updated bits. Then we have templating running on the client device, turning [?] into DOM.
T: ?
D: The cache is [?] to the client. The trick is having the right granularity. If you always assemble the page, then redundant parts in cache. Somewhere in between there's a good balance.
T: The more you break it up, the more invalidation events you have.
D: The granularity is starting point. Where Do you render HTML on the client side for clients who cannot assemble.
?: Do we want to keep this [existing pink stickies] e.g. backend service?
D: How do we render all these things. If its aAPI first, then there's a layer above that.
R: Are we further developing this existing
F: Re-architecting all these things is so much work
R: What's the motivation to potentially re-architect?
D: What are we trying to fix?
M: Was the redundancy of knowledge through html and mobile apps,
D: Currently maintaining 3 services.
T: That doesn't make sense. There are all separate types of client. We're going to continue to serve them all. We need to look at ways to share certain things between them.
C: We need to minimize the unique needs.
D: or expose everything through APIs and data. People are likely to build specialized clients. And if we have these apis we should use them.
Expose all the functionality you can use as an api, eg gadget people can use them.
T: we already have an API for almost everything. WE just don't have a combination for all.
C: Some clients have unique APIs when perhaps they shouldn't.
M: Posible motivation If someone wants to use our APIs in an ecosystem where php isn't

Supported.

D: If someone [?] without touching all the code for all the special pages, Sits on top of api layer. T
T: MW has components that can be shared between [php and javascript?]
D: Do we call the 2 together "MW"?
D: Somewhere at client end there's templating or [?]
F: ? D: For pagecontent yes, for everything else no
T: So we're moving parsoid back into php, my concern is mobilecontentservice outside MW and discover all the same problems
D: Core output of special page still generates html. Mobile Content generalized and baked into php.
?: So, we want to move the index.php (HTML output) layer further up the layer?
T: There's MW classes, and api php and index php as wrapper for html classes
D: Index php sits on top of that, generate a complete HTML page.
T: Big missing component is caching. System designed around caching arch.

SUMMARY:

D: Mostly inventory of what already exists and what we're trying to achieve.
Parameters for key aspects we came up with
Think about page content and how to split it up, e.g. infobox, specific kinds of content, wikidata, and non-structure html.
Aux things need to be turned into usable things, e.g. page history, or user rights info, this is a data interface and needs to be turned into UI somewhere somehow. We have much that gens HTML … we want some interface that can serve chunks of HTML, network edge or client or elsewhere, should be well defined for doing that. [?] already does this and could be expanded.
Currently hard to share code between the different experiences. If we want to change how certain elements look in special pages, you have to change them all individually. Can do it with css but not beyond that. Would be nice to have that flexibility. How much do want to invest to get there.
How important is it to cater to 3rd party, or for them to build UIs to get our functionality. It's unlikely they will build something that will cover all the funct we have, however they might build specialized interfaces, and it should be easier for them to do so.
We don't want redundant api implementations. We currently have 2 apis with some functions duplicated.
We have a lot of data in caches and on the wire. All the chrome is sent over and over, not ideal.
Caching is hard. The way clients interact with caching [?].
C: Is that like the Turbolinks thing?
Joaquin: [rendering on client] for partials.
Erik: ?
D: in my dreams that basically is MW, and serving static webpages isn't
Tim: caching is often unconsidered. Marvin was breaking caching for no good reason. Taking [?apart] and putting them back together in json. Instead of metadata encoded in json can have it encoded in html. Don't want to duplicate caches. Don't even want 2 copies of all the html for the site. Currently have multiple copies. Costs a lot to have a hot cache of so many duplicates. {?] but requires endpoints that can serve thousands of caches a second.
Corey: Brandon has input on this idea, and maybe caches aren't as much of a problem?
J: Restbase and usage, varies, if we invalidate them as they change, or [?] huge amount of variety on those things. Still have to store structured data for some things and html for others. Partial level, WE need to be careful about what we cache and where. Already come up in restbase and some ideas about that.

GROUP 2 - Participants (optional): John Bennett, ...

Discussion:

When we say html what are we saying? There is a page with user interface or markup of txt?
Would like to see a single way to do something and have somethng over the top it which is currently impossible on the mobile apps.
Things behave differently and all UI could be driven by api
Maybe work toward using action api for all the wiki data centric things -- Have the action api call out to a new api we haven’t yet created but is action api even the right abstraction?
Can we replace special pages with other special pages that look and act the same but use an api?
Server side rendering but using a consistent internal and external method
Action api as it exists today is not effective. Action api is not the way forward.
All of this behavior should be available everywhere
The goal was to provide the same functionality -- that everybody should be going thru the same interface into mediawiki via the api --that all clients should be equal

SUMMARY:

Erik: action api as it exists today isn't effective. Pretty sure we want a new api
T: in php?
E: interfaces behave diff, e.g mobileweb vs - should be unified.
E: All clients should be equal
Jon; some tradeoffs, the interface doesn't have to be equal, there are some places we can't support the same functionality. non-JS browsers are shrinking part of globe. Base of pyramid concept - most high-impact folks are poorest at base of pyramid, but also hardest to reach. We're going from "have to have desktop" to "have to have mobile phone, but smartphone" - i think it's ok to say "have to have JS to do some [advanced] things". 3 tiers, Reading, Editing, Moderating. With Reading obviously needing to support non-js
E: new api suggested for
Adam: wikidata-first api. The worlds can be separated quite nicely.
Bryan: remember that 50% of edits from action api.
ADamS: how wikidata uses action api isn;t ideal.
AdamB: [?]

GROUP 3 - Participants (optional): Bryan D, Leszek, Ian, Zack, Sage, Joaquinm, Dmitry, Alexia

Discussion:

Let’s just punt on whatever is below the API layer. There’s some kind of storage, some kind of business logic, we’re ignoring it for right now.
Does conversion of data to HTML have to happen server side?
- Might want to manipulate things later, even if there’s a baseline HTML that is delivered, we might want to do something on top of it. EG Sage is getting WikiText, adding spans, returning it to be parsed, and then applying styling to the return.
AB: state of the world
- Mobile Web is working to expose some Special Pages
- Moving to Mobile Content Service, delivers HTML to the client that can be rendered in the web view. May have additional data appended that allows for further enrichment
Precondition: there is a parser that works
What MUST be able to be delivered as HTML?
What SHOULD be able to be delivered as HTML?
What COULD be able to be delivered as HTML?
Competing approaches: do we compose after the fact (eg, Page Composition Service), or do we do everything in PHP?
Not going to do isomorphic rendering at this point. Could do a rich client experience but likely to only be bespoke for the moment.
Same URL routing based on structured data. Can render on the server side or deliver to the client.
Wikidata - looking at moving to external rendering (Node?), with the resulting HTML being inserted in to the page and delivered to the client

SUMMARY:

J: Bryan wrote current concerns and shuffled around. Assuming this existed and covered all use cases was uncertain. [? layers]
We understand this would be about having the sort of isomorphic serverside and clientside services. Many tradeoffs. 3rd party wikis [?]. Also wikis and other use-cases. Not everything needs to be one to one.
Approaches, same URL routing can work on server and client, same experience on both. [?] rendering the php and JS side, config variables and things per page injected, make navigating to other browser very different. Could have 2 rendering paths, one html only, main features should be reading editing watchlist diff etc. One top of that JS experience with new things like clientside navigation . [?] if you render from html, html only users get a great experience, but also [? downside].
If you want to provide different experiences,
Other approach, something like Turbolinks. Render happens on server, outputs HTML, JS injected for newpage. Rendering happens on server so not as much duplicated. Just for different parts of the page.
Too many products and use-cases, e.g. wikidata has own redenering service on node, and same thing for mobile apps [?]