Reading/Web/Projects/Invest in the MobileFrontend & MinervaNeue frontend architecture

This page pitches a project to invest in MediaWiki's mobile sites frontend which is currently served by the MobileFrontend extension and MinervaNeue skin.

Summary

• Phabricator task: https://phabricator.wikimedia.org/T195473

The mobile website (specifically MobileFrontend and MinervaNeue codebases) is technologically dated. This impedes development of new features and maintenance of existing ones. We propose a solution to, and have provided proof of concepts for, iteratively refactoring and revising the existing code in order to achieve the following outputs:

• Enable compressed asset bundling.
• Speed up unit tests and increase code coverage.
• Complete the MinervaNeue migration.
• Review and refactor components - separate presentation and logic.

In addition to solving the stated problems, we expect the completion of these tasks to yield the following outcomes:

• UI feature work is more independent of MediaWiki implementation decisions.
• Greater capacity, and throughput.
• Increased interest, engagement, and collaboration of internal and external contributors.
• Decreased developer onboarding time and enhanced understanding of code.
• Improved data savings and performance.
• Code sharing (pooled features and talent, reduced bugs and maintenance, gained stability).
• Significantly faster and better automated testing, fewer regressions.
• Fewer dependencies on WMF maintained third-party code.

We believe that this initiative is the responsible choice for building on existing work and a stepping stone to future platform evolution.

Background

Given the existing frontend infrastructure and the state of our mobile web frontend code bases, working on frontend (UI features) is challenging and painful in a variety of ways.

With the upcoming API focus from the Platform Evolution Program, the rapid innovation by browser vendors (e.g. Service Workers), and the learnings from Extension:Popups (Page previews) and the progressive web app prototypes, we believe we should invest more in our mobile web platform if we want to make future changes to mobile web, adapt to upcoming changes to MediaWiki Core, ease future maintenance, and have a good base for future feature work, whether based on new contributor or reader features.

This project is planned and budgeted in the annual plan, as technical work under the CDP Output 2.7. This document specifies and explains the work and its value, and sets measurable goals to track the progress on the outputs. The project will be worked on by the Readers Web team, with ~2 engineers focusing on this project for continuous amounts of time, leaving other FTEs to work on other goals.

The project is broken up in sub-projects (outputs), that are valuable in isolation, and are part of a step by step plan towards the larger goal. Each sub-project will be tackled sequentially & iteratively and on the production codebases (live websites), to ensure compatibility with the product, avoid divergence and minimise merge conflicts. We will not use big feature branches or require big unveil refactors.

The document follows the annual plan terminology, discussing first Outcomes that we want to accomplish, and then outlining the Outputs (sub-projects) that will be tackled to accomplish those outcomes.

Overview of outcomes

Given our previous work  and experience as a team, and the evolution of the industry and global FLOSS community tooling and ecosystem, we believe we can improve the state of the mobile web UI code bases, with the following outcomes:

1. Improve developer experience (onboarding, maintenance, development)
2. Reduce technical debt and make production improvements
3. Separate presentation and logic
4. Make Minerva and MobileFrontend better citizens of the MediaWiki ecosystem

See the Outcomes section for more detail.

Outcomes

1. Improve developer experience (on-boarding, maintenance, development)

Enable future use of tooling on the frontend code base to automate regression detection, improve asset sizes, and other productivity enablers.

There are many FLOSS tools, developed by the community and in wide use in the industry, that we can use to help development, maintenance, and preventing regressions.

We will be able toː

• Identify increases in asset sizes before shipping code
• Run our tests more often and faster
• Run our code through tools that make it easier to have cross-browser compatible code reducing overhead and bugs
• Use other tools that reduce bundle sizes in the most aggressive way to get the smallest possible size
• Automatically sort and bundle our frontend assets without manually specifying order, removing race conditions, and making it easier to create and refactor files to make new features or improve existing ones

Speed up development workflows by improving the unit testing process and improve MediaWiki performance in development by decreasing the load on ResourceLoader.

Because of the amount of files and the architecture of the QUnit browser tests, which run in the browser and require a full server roundtrip and ResourceLoader parsing on-demand, the feedback cycle in development for writing and running code and tests is detrimental to properly testing the code and running changes incrementally.

We will enable developers to run isolated unit tests quickly from the command line without having to spin up a server, shortening the feedback cycle and making it easier to write tests and use TDD processes when coding.

We will also remove the load and overhead we put into ResourceLoader by way of having too many modules and files which hurt performance, by bundling and grouping the files with tooling so that the ResourceLoader definitions stay at a minimum and the file order and file bundling into resource modules is automatic and resilient, reducing production bugs and developer overhead and freeing developers to work with and maintain the code more efficiently.

Smoother engineer on-boarding and contributions from the current and next generation of technical volunteers and employees.

Our code will meet expectations of newer developers (hires/volunteers) and be more pleasurable to work with by the team as a whole. This is hard to measure but intuitively valuable and likely to increase throughput of changes.

Currently, we use bespoke code and don’t use efficient tooling. Our code hasn't kept up with the changes through the years. This confuses and slows down the on-boarding process for many of our devs.

Measuring success

Some aspects mentioned above, like on-boarding, are hard to measure. Instead, we plan to measure proxies:

• We will add the following features to the development processesː
  • Asset size and performance budget checks for the frontend code are set and running in CI jobs per patch
  • Cross browser prefixing and modern Less compiling is set and configured with our supported browsers
  • Frontend assets specify their dependencies in the code via standard module systems and are bundled for ResourceLoader to serve
  • CLI based QUnit test runner is set up and running in CI jobs per patch
  • CLI unit tests can run isolated without having to spin up a server
• We will focus on writing developer documentation that new developers can run through with ease
• Lines of configuration for JavaScript file order will be removed and assets percentage of files (LOC) moved to use automatic dependency management
• Unit tests are migrated to the CLI runner for faster runs and feedback, we will measure the percentage of test files moved to the new test folder and start measuring code coverage and report on it

2. Reduce technical debt and make production improvements

Stop using unmaintained libraries & technologies (like hogan).

Hogan.js is unmaintained and abandoned, and we should stop relying on it when it makes sense and rely on better-supported alternatives. Using abandoned code is a liability for production software, even if it is very well battle tested.

Data savings and possible performance improvements from reducing the size of our frontend assets (JavaScript and templates mainly).

We will improve bytes shipped with minimal effort by relying on tooling.

By using better tooling we can automate many performance improvements that right now are very difficult e.g. tree shaking/better minification, using templates without a runtime payload/compute cost.

Measuring success

We will monitor the JavaScript bundle size closely and report the change after implementing the new tooling and minifiers.

Templates will be run in production without parsing/compiling overhead. We will report on the percentage (LOC based) of migrated templates that don't have the runtime overhead.

3. Separate presentation and logic

Decrease the complexity for new features to be developed and reusing existing components by making our code saner to work with.

When the mobile site was built, its component system used an inheritance model with templates and template partials. Projects like React have shown that the alternative of declarative rendering and a component based composition can make development faster. In addition, libraries such as Hogan are no longer maintained, suggesting their days are over.

Today, we have overlays for all our components - TalkOverlay, EditorOverlay and ImageOverlay. We then have base components such as TalkOverlayBase, EditorOverlayBase and Overlay. As a result, it’s very difficult to follow our code and understand how each class operates without looking at ancestor classes. This makes testing and new component creation harder as many of our classes are hard-wired for specific but similar workflows. Creating a new overlay usually requires creating templates and JavaScript classes and lots of additional wiring. We envision a future where if a designer wants to create a new Overlay, a developer will find this incredibly easy - e.g. 5 lines of code rather than several files.

Measuring success

• We will be able to tie the MobileFrontend code to the style guide and identify missing components.
• We will remove most inheritance chains in classes in favor of composition.

Increase the quality of our code, reducing the numbers of bugs that make it to production.

Much of our JavaScript has no tests due to the way it was architected and built (several designers/deadline driven work). A re-org/refactor of our code provides the opportunity to write unit tests as we go. This will help us understand our code; identify edge cases and identify areas we can simplify.

Many features built by volunteers are not production ready (e.g., Categories) - we will improve/simplify/remove redundant code in the process.

Measuring success

We will provide baselines of test code coverage before and after the refactoring.

Establish clear boundaries from our mobile web frontend and MediaWiki and enable reuse of the mobile web UIs and workflows outside MediaWiki (apps, existing or new products, other projects in the ecosystem, etc).

Right now, our mobile site is too tightly coupled to MediaWiki JavaScript, the Hogan templating library, jQuery and other large libraries of code. We believe that by outlining and abstracting where our dependencies are, we will be capable of breaking free of these dependencies in the future with minimal effort (in case we have to respond to unexpected changes in our ecosystem). This also opens up opportunities to share code with Android and iOS (e.g., the Page Library).

Measuring success

By looking into the % of code (LOC based) that has been migrated to a library and that can be run without MediaWiki.

4. Make Minerva and MobileFrontend better citizens of the MediaWiki ecosystem

This project will make developers’ lives easier, increase awareness of cross-skin/mobile testing by other teams, and reduce grumbles about the structure and function of MobileFrontend and Minerva in the MediaWiki ecosystem.

Since we split MobileFrontend from Minerva we have seen increased attention and efforts in testing. For instance, some Foundation teams frequently test against the Minerva skin as part of their QA process now (whereas they didn’t before), and there has been renewed interest in Minerva/MobileFrontend. This move helped people conceptualise the different purposes of these 2 repositories. It also guided engineers on where code should live. That said, there is still a little confusion with the setup:

• On the code level with regards to where code should go - where code lives, code review and feature development is a little rougher than it needs to be.
• Editors/third parties have expressed an interest in using Minerva as their mobile AND desktop skin, which is currently a buggy experience. This work will fix those bugs and make Minerva a viable alternative alongside Vector and other skins. It will also improve the overall experience of the Minerva skin.

Right now, MobileFrontend and Minerva are coupled due to their JavaScript and other UI assets. By turning the JavaScript in MobileFrontend into a library inside Minerva, we will allow people to run MobileFrontend without Minerva with an additional build step (we envision MobileFrontend having an optional install step for people who want to do this). Currently, it is quite painful even for our own developers to build features as they have to consider 2 repositories and write patches that depend on each other inside MobileFrontend and Minerva. Having the UI code in Minerva will take much of this pain away. For development of the mobile site within WMF, we will only need to work inside one repository. Third parties who want different setups will be able to have them by following specified build steps (e.g. MobileFrontend wikis without Minerva).

Measuring success

It will be possible to install Minerva without MobileFrontend (and vice versa). This will be well-documented and the developer community will be notified.

Outputs - Subprojects

Automate asset bundling

The mobile web projects have very extensive manual configuration for the frontend assets and specifying their dependencies. Just the configuration for MobileFrontend has +1000 lines of code and Minerva has ~500 lines of code.

This current state has a negative impact on development practices, ResourceLoader performance, general development, and our ability to change, move and split the code.

In support of

• Outputs:
  • Speed up unit test execution and increase code coverage
  • Review and refactor components

• Outcomes:
  • 1. Improve developer experience (onboarding, maintenance, development)
  • 3. Separate presentation and logic

Proposal

Use a Node.js based bundler (webpack) to compile the JavaScript files into different entry points automatically based only on source code import statements. These bundled files will then be given to ResourceLoader for serving.

This approach is the same one taken in Extension:Popups (Page Previews), which has proven to be an effective strategy.

We will also use UglifyJS for minifying our JavaScript files for production, which should give us approximately 25% savings on file sizes for the mobile website.

Proof of concept

https://gerrit.wikimedia.org/r/#/q/topic:dreamofwebpackification+(status:open+OR+status:merged+OR:status:abandoned)

Measuring progress

See Invest in the MobileFrontend & MinervaNeue frontend architecture/Progress.

Speed up unit test execution and increase code coverage

Tests are a crucial part in the development flow to verify that the code works as intended and is future-proof against changes of behavior. With the current stack tests take a long time to run (because of the ResourceLoader strain and the overhead of running through MediaWiki). As a result of the cost of running tests and the slow feedback cycle during the development flow, we haven’t kept up with properly testing the code, and this has negative effects on feature stability and resilience, and on confidence in changing the code.

In support of

• Outputs:
  • Review and refactor components

• Outcomes:
  • 1. Improve developer experience (onboarding, maintenance, development)
  • 2. Reduce technical debt and make production improvements
  • 3. Separate presentation and logic

Proposal

Leveraging the Automatic bundling goal, we will follow the Extension:Popups approach and migrate as many unit tests as we easily can to run in the CLI (Node.js based). Tests that make extensive use of MediaWiki or are too tied to ResourceLoader will be considered integration tests and kept in mediawiki-qunit.

Untested files simple enough to test will have new tests added in the process of migration for Automatic bundling.

Proof of concept

https://gerrit.wikimedia.org/r/#/q/topic:dreamofwebpackification+(status:open+OR+status:merged)

Example

https://gerrit.wikimedia.org/r/#/c/406783/

Measuring progress

See Invest in the MobileFrontend & MinervaNeue frontend architecture/Progress.

Make Minerva independent of MobileFrontend

In support of

• Outcomes:
  • 3. Separate presentation and logic
  • 4. Make Minerva and MobileFrontend better citizens of the MediaWiki ecosystem

Proposal

3rd party support is important to the MediaWiki community and causes the team extra maintenance effort (dealing with talk page questions and bug reports). Currently there is an implicit relationship where Minerva requires installing and configuring MobileFrontend, and MobileFrontend requires Minerva. As such, using only MobileFrontend (with Vector for example) or using only the skin Minerva (on desktop) is complicated and causes many issues to MediaWiki administrators.

We will make Minerva an independent skin that can easily be used independently, like other skins. We will continue to support mobile-friendly special pages inside MobileFrontend and clarify the purpose of MobileFrontend and the function it serves (which currently causes a lot of confusion).

We will make MobileFrontend use a UI library for the mobile-friendly Special pages provided. Minerva will also use the same mobile-friendly UI library (which could be used by any skins or other projects in the future).

To do so, we will migrate all UI components and assets into a library, which will be stored in the Minerva skin code repository for development convenience, but also published as an independent library.

Going forward, any engineer working on mobile web - Minerva or related features - will work in the Minerva skin repo. MobileFrontend will then effectively live in maintenance mode.

For WMF products, MobileFrontend will use the installed skin Minerva to get the UI library needed.

For 3rd parties who want to use MobileFrontend with another skin that’s not Minerva we will provide an additional install step for them to do so. In exchange for this slight discomfort they will have more control of their user experience. Websites not affiliated with the WMF (e.g. wikitravel) will have more reason not to inherit Wikipedia’s “brand”.

We believe this will reduce the amount of the work the team has to do in the arena of third party support (we get plenty of bug reports), freeing us to focus on products.

Implementation details may change, depending on activity in T107561 (how will MobileFrontend use Minerva’s components/vice versa).

We’ll email wikitech-l and mobile-l to let MediaWiki community members know about this change, before start as a consultation, sending updates via email with the monthly readers email and after finishing presenting the results.

Proof of concept

N/A

Measuring progress

We will monitor the resources/assets in MobileFrontend and how they move to the MinervaNeue repository, (except the special pages sources or other files MobileFrontend specific ones).

Review and refactor components

In support of

• Outcomes:
  • 1. Improve developer experience (onboarding, maintenance, development)
  • 2. Reduce technical debt and make production improvements
  • 3. Separate presentation and logic

Proposal

We will re-evaluate what we need/want from this at this point. When the previous projects have been successful, and if time allows, we will generate a concrete proposal for next steps. Some goals for this would be:

• Separation of concerns, separating UI and network and business logic
• Making our code testable
• Using composition instead of inheritance in the UI components
• Declarative UI rendering

Proof of concept

N/A

Measuring progress

We will define this subproject further as we get closer. It is left open ended right now to allow us the opportunity to choose the approach best-suited to the foundation’s long term objectives and the team’s workload during the FY. In the long-term, we believe these changes will help tackle design debt, reduce technical debt, and simplify the UI code to ease onboarding and development of new features, amongst other things.