Wikimedia Engineering/Report/2014/October

Major news in October include:

the removal of support for SSL 3.0 on Wikimedia sites to protect users against the "POODLE" vulnerability;
the release of the first native “Nearby” functionality in the Android and iOS Wikipedia apps;
the launch of the Structured Data project, an initiative to augment Wikimedia Commons using the technology developed for Wikidata.

Engineering metrics in October:

150 unique committers contributed patchsets of code to MediaWiki.
About 27 shell requests were processed.

Upcoming events

There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.

For a more complete and up-to-date list, check out the Project:Calendar.

[edit or add events] [refresh table] [view credits]

Date	Type	Event	Contact
3 November 2014		Tech Talk: The MediaWiki Content Translation Extension 1800-1900 UTC in #wikimedia-office ^connect.	Joel Sahleen
14 November 2014–16 November 2014		Dutch GLAM and Wikidata hackathon (Amsterdam, Netherlands)

Personnel

Work with us

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

Announcements

Marcel Ruiz Forns joined the Analytics Development team (announcement).
Guillaume Paumier joined the Product and Strategy team as Senior Analyst (announcement).
Bartosz Dziewoński joined Wikimedia as a Features Contractor (announcement).
Moriel Schottlender converted to full time staff member status as a Software Engineer in Features Engineering (announcement).
Marielle Volz joined the Wikimedia Foundation as a Software Engineer for the Editing team (announcement).

Technical Operations

Dallas data center

In October we completed the initial stage of our data center deployment in Dallas; finishing the setup of essential infrastructure, and making sure all essential project data is being replicated and/or backed up in Dallas. The Ops team is now moving towards the next phase; making codfw capable of serving the wikis independently of eqiad, ideally using HHVM.

Tampa data center

On the first day of October we shutdown nearly all remaining servers in Tampa, allowing some room to resolve unexpected problems should they occur. On October 6, we started removing all equipment from their racks, and shipped equipment with remaining value to Dallas. By October 12, we had vacated the facility completely, marking the end of 10+ years of Wikipedia's home in Tampa.

Labs metrics in October:

Number of projects: 145
Number of instances: 413
Amount of RAM in use (in MBs): 1,992,192
Amount of allocated storage (in GBs): 20,395
Number of virtual CPUs in use: 975
Number of users: 4,109

Tool metrics:

Number of tools: 947
Number of tool maintainers: 550

Wikimedia Labs

Labs suffered a brief, partial outage on October 7 due to a presumed out-of-memory failure on a virtualization node. Everything was back up and running after about 30 minutes; Andrew rebalanced instance load to forestall similar such failures.

New hardware has been ordered to expand Labs VM capacity. With luck this will be in place within the next month or so.

Marc redesigned the default base image for labs instances. /var/log is now in its own partition, and we support some limited partition resizing.

Yuvi has made a good start on setting up instance state monitoring.

Marc added an Ubuntu Trusty bastion and exec node to Tool Labs. Some work is still in progress here due to package renames and dependency changes.

Marc wrote many pages of operator documentation for Tool Labs.

Features Engineering

Editor retention: Editing tools

VisualEditor [edit]

In October, the team working on VisualEditor improved some features, prepared the way for table structure editing, and fixed over 60 bugs and tickets.

The keyboard shortcuts for items in the toolbar's menus now show on the right hand side, to make them easier to find. We made a few improvements to the display and editing of templates. Firstly, template fields can now have a value automatically set, like today's date for clean-up notices. Links inside templates now show in red if their targets don't exist, like for the editing surface. Last month's change to warn against saving a template with missing required fields was improved by highlighting them, as is the warning about trying to make an image 0 pixels wide.

The initial tools to edit the structure of tables were finalised and will be made available next month. We made progress on providing a new auto-filled citations tool, and improvements to the link editing and media searching tools, all of which will be coming in the near future.

The deployed version of the code was updated five times in the regular release cycle (1.25-wmf3, 1.25-wmf4, 1.25-wmf5 and 1.25-wmf6).

Editing [edit]

In October, the Editing team worked on the front-end standardisation project, now one of Wikimedia Engineering's Q2 priorities, as well as the work on VisualEditor, both of which are reported separately. The team welcomed Marielle Volz as a new team member, who will focus on improving citations support. The team created the wikitext concept of <indicator>s, a way to show icons which tell the user something about the page in MediaWiki. These features are now supported in the Vector, Monobook, Cologne Blue and Modern skins, and will replace the need for the local CSS hacks that some wikis have used to approximate this feature. The TemplateData extension now wraps its contents in a <noinclude> tag unless editing a sub-page, and the help link's target can now be set to a page other than the one on MediaWiki. The last vestiges of the CiteThisPage extension were removed from the Cite extension, as part of a complex manoeuvre to simplify the naming of the system; it now has its own home. The team's more significant work, on front-end standardisation, is now reported under that project.

Parsoid [edit]

October has seen a lot of maintenance-type work in Parsoid land. We updated our logging infrastructure to send our event logs to LogStash, which makes it far simpler to identify any production errors. We also started a round of code cleanup and improving readability / maintainability to moving to Promises API which eliminates the callback mess common with node.js code. We were involved in debugging some Varnish-related issues that were causing excess timeouts than needed; the issue is now fixed, and we improved the efficiency of queued Parsoid parsing jobs. At the end of October, we also initiated an upgrade of the Parsoid production cluster from node 0.8 to node 0.10. In non-maintenance work, we continue to fix bugs and compatibility issues with the current default rendering. We've also been working to find entirely CSS-based solutions to citation customizations. This CSS-based approach will simplify customizability, and also free clients from having to worry about MediaWiki site messages. This is still work in progress as we are finding gaps and fixing them.

Core Features

Flow/Project information [edit]

In October, the French WP Forum des nouveaux offered a Flow board as an alternate way for new contributors to ask for help. The Forum is currently using both a wiki talk page and a Flow board; a goal for the team is to build all of the functionality that the Forum needs in order to switch over completely to a Flow board. The team also built a new modal dialog for moderation actions, put unread Echo notifications in reverse chronology, and worked on the back-end needed for creating a Search feature on a Flow board.

Front-end

Front-end libraries standardization

In October, the Front-End Standards Group continued their work to extend and adopt the OOjs UI library. This included the finalisation of the initial version of the MediaWiki theme in collaboration with the Design team, which can be explored in the online demo; this will be deployed into MediaWiki's use of OOUI in the next few weeks. The library now has a parallel, PHP language port. This will allow pages to be created server-side and later as a progressive enhancement made rich for JavaScript-supporting clients. This PHP port will be made available inside MediaWiki soon, and will be used to provide the framework for the future skin architecture. The plan is to demonstrate it in rebuilding a small number of the core interfaces into OOUI by the end of the calendar year. Other improvements to the library include support for toolbar menus to only show a few items initially with option to expand later, and showing keyboard shortcuts, expansion of the developer documentation, and fixing a number of bugs. Two new widgets were created – a widget for a button that's also an input, and a widget for a progress bar, which will be used as part of the work to convert the new PDF generation system to use the OOjs UI framework.

Mobile

Wikimedia Apps [edit]

In October, following on from the successful launch of Nearby on iOS in September, the Wikimedia Apps team released Nearby on Android as well. Nearby allows you to see articles about things that are near you, and read their articles to learn more about them. The team also released features that roll up disambiguation and page issues into a button underneath the page title, so that the content of the article is brought closer to the top of the article. We also released a refactor of the technical architecture of the app to bring the app closer to compliance with Google's standards for Android apps, which also brought a few minor UX improvements. On the iOS side, a bug fix build was released which improved the UI scaling on iPads, fixed a few minor issues with the table of contents, and fixed miscellaneous crashes. At the end of the month, a build was submitted to Apple which reworked the way the app retrieves information from the servers, which should increase the stability of the app and reduce crashes.

Wikipedia Zero [edit]

During October 2014, the Partners engineering team enhanced the partners portal (ZeroPortal) extension to improve security and maintainability, added better font support for the Wikipedia Zero end user experience, migrated the remainder of operator partner configurations to use the unified design in support of lowered cache fragmentation on the mobile web Wikipedias, improved the fidelity of zero-rated Graph extension pageview statistics relative to the legacy Limn graphs while streamlining batch processing, made Wikipedia Zero end user information screen overlays follow MobileFrontend style, created Cucumber tests for the partners portal, added colorized crouton support to the Android app while reducing intrusive warnings on the Android and iOS apps for zero-rating (colorized iOS banner code also merged, will be in forthcoming release), prepared analysis and code for language-aware redirects from the m.wikipedia.org webroot, consulted with partners on API usage and app preloads and HTTP header enrichment and language aware redirects, and examined the interplay of Wikipedia Zero with CentralNotice (e.g., fundraising) banners; currently, CentralNotice is suppressed when Wikipedia Zero page rewriting is in force, but in the future it would be nice to show non-fundraising CentralNotice content.

Mobile web projects [edit]

This month the mobile web team created a new experimental interface for WikiGrok, a feature designed to provide an easy way to contribute structured data to Wikimedia projects. We are currently showing the two different versions of WikiGrok to readers and logged in users on the mobile beta site on English Wikipedia and gathering qualitative and quantitative data on engagement with the feature and quality of contributions. We also focused on backend integration to allow us to generate Wikidata questions on more articles, in preparation for testing at scale on the stable mobile site in the coming months.

Language Engineering

Language tools [edit]

CLDR version 26 updates that affected plural rules for MediaWiki i18n have been integrated. The affected messages have been updated and translators have been informed. As part of regular maintenance of the Translate extension, many unused configuration variables have been removed and Special:Translate now works better with custom skins. Niklas also fixed a major regression in a refactoring which broke the message group selector.

Content translation [edit]

The second version was deployed, and Kartik worked with the Operations team and upstream developers of Apertium to prepare requisite packages for the machine translation service. The category adaptation was added, as well as bi-directional machine translation support between Catalan, Portuguese and Spanish. Language support was extended, and development started on the translation dashboard and the 3rd version of Content Translation. The first draft of the graduating language support specification was completed; This specification will guide selection of further language pairs to be supported through the Content Translation tool.

Platform Engineering

MediaWiki Core

Search [edit]

In October we prepared for November in which we deployed Cirrus to all the remaining wikis by installing new servers installing new versions of Elasticsearch and our plugins. We also fixed up regex search which had caused a search outage.

SUL finalisation [edit]

The major engineering development work for single-user login wrapped up this month and there are now three tools being tested on beta labs.

Special:GlobalRenameRequest is the form that users who are renamed post-finalization can fill out to request a new username and submit it to Stewards and global renamers. Special:GlobalRenameQueue allows Stewards and global renamers to manage rename requests in a simplified interface form for post-finalization requests to be sent to from Special:GlobalRenameRequest. Lastly, Special:GlobalUserMerge is in very early testing stages. This tool will allow Stewards to merge the accounts of two or more usernames owned by one person who may have multiple account names due to conflicting usernames.

Once these tools are tested out and deployed, outreach will begin by talk page contact and by email where possible with all accounts that will be renamed due to SUL finalization.

Library infrastructure for MediaWiki [edit]

The project kicked off mid-month with the merge of patches that enable PSR-3-based logging by MediaWiki. These changes are being tested in beta and will begin to roll out to the production cluster in early November with the 1.25wmf6 release branch.

An investigation into the possibility of using a package manager for JavaScript libraries in MediaWiki closed with the consensus opinion that we are not ready to choose a package manager at this time. The frontend standards group will revisit this decision in three to six months. It was agreed however that as far as possible JavaScript libraries should follow the guidelines for library development that are being worked on for PHP code.

Initial work has begun on a Profiler implementation that uses XHProf to collect information about the runtime costs of MediaWiki code. This approach to profiling will enable collection of information on code running on Wikimedia servers without relying on explicit wfProfileIn() and wfProfileOut() calls. This in turn will make splitting code out of MediaWiki core easier.

Security auditing and response [edit]

We completed security reviews for WikiGrok, Labeled Section Transclusion headers, the IEG grant-review application, and RecentActivityFeed. We also released security updates for CentralAuth and MobileFrontend.

Release Engineering

Quality Assurance [edit]

This month we welcomed Elena Tonkovidova to the WMF staff as our new QA Tester. Elena will be working closely with Rummana Yasmeen in that role. In support of our QA efforts, we invited Elisabeth Hendrickson, author of the book "Explore It!", to a well-attended meetup at the WMF office for a presentation about Exploratory Testing. Other projects for the month include adding a Ruby style checker Rubocop to our Jenkins builds, creating a Ruby style guide, further enhancements to the Vagrant development environments, updates to the beta labs test environment, and other projects.

Quality Assurance/Browser testing [edit]

This month we began a significant project for improving and refactoring existing browser tests. Starting with the Echo and Flow repositories, we intend to update and improve the browser tests across all the repos to be more modern, flexible, maintainable, and reliable. Our new Ruby style checker Rubocop in all of the Jenkins builds will help greatly in this effort, as will major improvements to the architectures of the features being tested, of which VisualEditor is a notable example.

Multimedia

Multimedia [edit]

In October 2014, the multimedia team started work on the Structured Data project, co-hosting a weeklong bootcamp in Berlin with community members and the Wikidata team. Participants developed first ideas for data models and user interface designs, as well as a high-end API prototype to read and write machine-readable data on Wikimedia Commons. These first ideas are now being documented here on Commons and in these project slides, for discussion purposes. If you would like to get involved, we invite you to sign-up for the newsletter and join the discussions on the Structured data hub. You can also learn more about the related metadata cleanup drive in this blog post. Our next office hours chat on Structured Data will take place on Thursday, November 20 at 19:30 UTC, on #wikimedia-office ^connect (Freenode IRC).

The team also developed and released more improvements to Media Viewer, based on feedback from the recent community consultation and user research. These improvements include: an easier way to disable Media Viewer for personal use, as well as re-enable it from a file page, pre-render thumbnails on backend for new uploads and make MediaViewer text larger in Monobook. Our last improvement for this release will be a caption or description right below the image, to be launched in November. In parallel, we also continued our code refactoring for the UploadWizard, and are now analyzing metrics for the upload funnel analysis, to gradually fix the most critical bugs that cause error messages during upload.

Lastly, the team hosted a quarterly project review for multimedia and planned its activities for this quarter, as shown in these slides. For more information about our work, join the multimedia mailing list.

Engineering Community Team

Bug management [edit]

A very active bugday on Offline content generation / PDF rendering took place, organized and led by User:Nemo bis. Wikimedia Bugzilla received several security updates (related RT tickets 1 and 2). A new component for IEG was created. Andre cleaned up some older Backport flags and removed the "Mysterious future" Target Milestone in the Wikimedia product.

Phabricator/Migration [edit]

After LDAP and SUL were successfully set up, the registration was enabled for everybody on phabricator.wikimedia.org. Bugzilla users are encouraged to create an account in Phabricator. Phabricator now allows interacting with external (non-Phabricator) users via email (like provided by RT). The Phabricator preview instance was set up on https://bugzillapreview.wmflabs.org/ to show how tasks migrated from Bugzilla look like. After announcing, the community provided valuable feedback in the bugzilla-preview project. Chase and Mukunda worked on fixing many reported and outstanding issues. Mukunda continued to work on the security extension, access to private tasks for their authors and redirecting old bugzilla.wikimedia.org URLs. Chase finalized the logic to import Bugzilla data into Phabricator and handling Bugzilla comment metadata. QChris made Gerrit create notifications in Phabricator tasks about related patches in Gerrit. Christopher implemented a free alternative to Google Charts for the future Phabricator burndown charts. Phabricator now sends mail to the wikibugs-l mailing list for all task updates, similar to Bugzilla. Regarding on-wiki templates, Helder updated the 'tracked' template to accept Phabricator parameters and Quim ported the 'Extension' template to link to Phabricator in its Bugzilla section. Furthermore, Phabricator user profiles show which MediaWiki.org account a Phabricator account is associated with and have an IRC alias field. Entering a real name is now optional. Andre informed users of Tags in Bugzilla that they will be removed. Quim created project creation guidelines and started discussions to plan migrating code review from Gerrit to Phabricator.

Mentorship programs [edit]

Wikimedia announced its participation in the 9th round of the FOSS Outreach Program for Women. We started the selection process with 15 applications received. We started preparing Wikimedia's proposal for Google Code-in. We participated in the Google Summer of Code Reunion, a special summit celebrating 10 years of this pioneering program.

Volunteer coordination and outreach [edit]

October was a month full of events. We organized a Wikimedia Tech meetup in San Francisco about Exploratory Testing for Complex Software; Lessons from Cloud Foundry, as well as two Tech Talks discussing The Dashboarding Problem, and Design Research in Product Development. The preparations for the MediaWiki Developer Summit 2015 continued, and we decided on all travel sponsorship requests.

Analytics

Analytics/Research and Data [edit]

This month we started recruiting a full-time research analyst dedicated to traffic and readership research and analytics.

We created a number of reports on mobile readership, including: daily snapshots of pageviews by device class and access method; a report on OS version breakdown for iOS app users and overall app usage; updated stats on the impact of the tablet redirect.

We started the documentation on mobile microcontributions and provided instrumentation requirements and experimental design support for a series of tests planned by the Mobile team to be rolled out in this quarter. We also started background research on missing claims in Wikidata as a function of item classes and language, to help inform the design of new types of microcontributions.

We hosted our monthly research showcase, with a presentation by Aaron Halfaker on Wikipedia as a socio-technical system and a guest talk by BarcelonaMedia's David Laniado on sentiment analysis of editor discussions.

We made substantial progress on defining reader engagement metrics. We also supported several external requests for data, including a request to extract a complete set of references with PubMed Identifiers and a dump of revert metadata for the English Wikipedia.

We completed a preliminary analysis of HHVM's effect on newly registered editors.

Kiwix

The Kiwix project is funded and executed by Wikimedia CH.

We released a minor version (1.91) of Kiwix for Android fixing a few annoying issues. Preparation to welcome Google Code-in junior developers has started.

Wikidata

The Wikidata project is funded and executed by Wikimedia Deutschland.

In October the Wikidata team focused on performance, bugfixing and planning for structured data support for multimedia files. This included a week-long workshop with volunteers and the WMF multimedia team in Berlin. They made progress on statements on properties, simple queries and usage tracking and rolled out first partial changes to the user interface that are part of the new design. They answered questions in an office hour on IRC about structured data for multimedia files and are looking for input on how to improve references and constraints violation reports. And to top it off the project celebrated its second birthday.

Future

The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.