Wikimedia Engineering/Report/2013/November
Engineering metrics in November:
- 146 unique committers contributed patchsets of code to MediaWiki.
- The total number of unresolved commits went from around 1122 to about 1230.
- About 29 shell requests were processed.
Major news in November include:
- Beta Features, a new way for users to try out new features on Wikipedia and other Wikimedia sites before they are released for everyone;
- The launch of our search for a VP of Engineering;
- A retrospective by the Mobile engineering team on best practices for collaboration while working distributedly;
- The activation of OAuth on Wikimedia wikis, which allows users to authorize third-party applications to take actions on their behalf without sharing their password;
- A presentation of the "Wikidata concept cloud";
- A retrospective on the ability to add musical scores to pages on Wikimedia sites.
Note: We're also providing a shorter, simpler and translatable version of this report that does not assume specialized technical knowledge.
Upcoming events
[edit]There are many opportunities for you to get involved and contribute to MediaWiki and technical activities to improve Wikimedia sites, both for coders and contributors with other talents.
For a more complete and up-to-date list, check out the Project:Calendar.
Date | Type | Event | Contact |
---|---|---|---|
5 December 2013 | WMF Metrics and Activities meeting | Erik Möller | |
13 December 2013 | Tech Talk: MediaWiki Upgrades and Extension Dependencies – Ideas for Improvement | Mark Hershberger and Markus Glaser | |
17 December 2013 | Tel Aviv MediaWiki Winter Hackathon | Oren Bochman |
Personnel
[edit]Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.
- VP of Engineering
- Software Engineer - Growth
- Software Engineer - Core Features
- Software Engineer - VisualEditor (Features)
- Software Engineer - Language Engineering
- Software Engineer
- Senior Software Engineer - Team Lead
- QA Automation Engineer
- Software Engineer Data Analytics (Back End)
- Dev-Ops Engineer - SRE
- Graphic Design Interns - Paid
Announcements
[edit]- Jeff Hall joined the Platform engineering group as a member of the QA team (announcement).
- Aaron Arcos joined the Platform engineering group as a volunteer developer working with the Multimedia team (announcement).
- Dario Taraborelli was promoted to the position of Senior Research Scientist, Research and Data team lead. (announcement).
- Aaron Halfaker was promoted to the position of Research Scientist (announcement).
- Moiz Syed joined the User Experience team as User Experience Designer (announcement).
Technical Operations
[edit]- A new dynamic proxy system has been deployed on Labs; it allows the admin of any project to arrange for public web access and a dedicated DNS hostname for a project instance without requesting an IP address. Labs staff and volunteers will now be reclaiming quite a few IPs as existing projects migrate to a dynamic proxy setup.
- The WMF has hired a short-term contractor, Mike Hoover, to assist with the migration of Labs infrastructure from Tampa to our new datacenter in Ashburn. Mike has spent a lot of time exploring the existing infrastructure and running test setups; soon he will start to configure the new OpenStack nodes in production.
- Andrew Bogott has been working on cleaning up stale and unused resources. He's working on some automatic documentation that will help users track the status of their projects and instances with an eye towards predicting the impact of the coming migration.
- Labs suffered two brief outages: a brief, self-inflicted network failure, and a longer outage during which one of the virtualization hosts failed. Both outages were swiftly resolved, but there's a bit of lag as some tools and services failed to come back properly afterwards due to poor distribution of virtual servers (inter alia, both the grid master and the shadow [backup] master were on the same server).
- Preparation for the move to the Ashburn data center is well in progress, with the new storage server being physically configured this week as well as the new hardware servers for user databases (including the new PostgreSQL instance intended for OpenStreetMap).
Editor retention: Editing tools
[edit]After these issues were sorted out, we continued with continuous improvement and fixes. Editing support for magic words and categories was improved, several dirty diff issues were fixed and the API was refined for page-independent wt2html and html2wt conversion. See our deployment page for details.
Cassandra load testing for the Rashomon storage service continued and uncovered several issues that were reported back upstream. With Cassandra 2.0.3 the 2.0 branch is now stabilizing in time to make deployment in December feasible. Cassandra is now stable at extremely high write loads of around 900 revisions per second, which is more than 10 times the load we experience in production.Core Features
[edit]Flow Portal/Project information
Growth
[edit]The team also spent significant time on the research and design preparations for its anonymous editor acquisition and Wikipedia article creation projects. This included participating in a community Request for Comment about a potential Draft namespace for articles, requirements gathering, and working on a Draft namespace patch.
Matthew Flaschen and Pau Giner attended the Wikimedia Diversity Conference and presented (along with Jared Zimmerman and Vibha Bamba) on how diversity related to the team's engineering and product work.Support
[edit]- Team highlights for this past month include a very successful Open Source Language Summit in Pune, India co-organized with Red Hat. More than 60 developers joined in to collaborate and work together on improving language support for Wikipedia on the web and mobile. Work sprints on integration of input methods in VisualEditor, Indic Fontbook specification, mobile input methods and content translation were held.
- The team also fixed and deployed several issues related to performance and to saving preferences for the Universal Language Selector (ULS). Other tasks completed include creating a class for interlanguage links using where the Autonym font can be used only for autonym items. The team also worked on collating documentation about all initial inclusion requests for each web font served through ULS also documented in the font.ini files of each font in the repository.
MediaWiki Core
[edit]Site performance and architecture
Security auditing and response
Quality assurance
[edit]Quality Assurance/Browser testing
In November, the Engineering community team held their second monthly showcase, as well as their quarterly review for the July–September period.
- Anu G Enchackal - UploadWizard:OSM Map Embedding (mentored by Gergő Tisza)
- Diwanshi Pandey - Complete the MediaWiki development course at Codecademy (Yuri Astrakhan)
- Brena Monteiro - mediawiki.org homepage redesign (Heather Walls and Quim Gil)
- Be Birchall - Clean up Parsoid round-trip testing UI, including using a templating system (Marc Ordinas and Subramanya Sastry)
- Maria Pacana - Clean up tracing/debugging/logging inside Parsoid (Subramanya Sastry and Arlo Breault)
- Niharika Kohli - Compact interlanguage links as a beta feature (Sucheta Ghoshal and Pau Giner)
Volunteer coordination and outreach
Multimedia
[edit]We also released Beta Features on all Wikimedia wikis, where it is already used by thousands of users. This experimental program invites users to try out new features before they are released widely, then give feedback to developers. To use Beta Features, click on the small 'Beta' link next to your 'Preferences' on your site, or test the latest version on MediaWiki.org. Fabrice Florin managed product development, led the creation of the Multimedia Vision 2016 (with Pau Giner), hosted roundtable discussions and updated the team's multimedia plans, based on community and team feedback. Bryan Davis, Aaron Schulz and Chris Steipp reviewed new code for the upcoming GLAM Toolset for batch uploads by museum curators. We also welcomed Aaron Arcos as volunteer software engineer, who is joining our multimedia team full-time through Spring 2014.
To discuss these features and keep up with our work, we invite you to join the multimedia mailing list. We are also recruiting for a senior software engineer position on our team.In collaboration with the Platform team, we ran an A/B test to determine performance gains of localStorage. The results indicate that the use of localStorage significantly improves the site's performance for the end user: Module storage is faster. Readers whose pages load slower tend to browse less. Mobile browsers don't seem to benefit substantially from caching.
We published the results of a test designed to explore if displaying a short tutorial could improve the first-edit completion rate of newly-registered users on mobile devices. The results support the hypothesis, indicating that edit guiders are a good onboarding strategy for new mobile users.
We ran an analysis of anonymous editor acquisition as background research for new onboarding strategies designed by the Growth team and found that editors who edit as an IP right before registering an account are our most productive newcomers.
On November 9, 2013 we hosted the inaugural Labs2 Wiki Research Hackathon: it was the first in a series of global events meant to "facilitate problem solving, discovery and innovation with the use of open data and open-source tools" (read the full announcement). Highlights from the event are available in the latest issue of the Research Newsletter. We are planning to host a new hackathon in Spring 2014 and we are actively seeking volunteers to host local and virtual meetups.The Kiwix project is funded and executed by Wikimedia CH.
- We have released two new versions of Kiwix for Android this month (1.5 & 1.6), providing many new features; most of them were developed by young new developers as part of the Google Code-in program. We have also released a new and unique tool to easily create ZIM file yourself from data on your hard drive; the tool is stable and can now be used. Work continues around tools based on Parsoid output, especially as we need to rewrite the ZIM-related code for the MediaWiki offline toolchain, currently under heavy re-engineering.
The Wikidata project is funded and executed by Wikimedia Deutschland.
- Wikidata developers held an office hour to give a status update and answer questions (read the log). In addition, they worked on ranks, ordering of statements and the quantities datatype. The quantities datatype is needed, for example, to enter the number of inhabitants of a country in Wikidata. It is available for testing now on http://test.wikidata.org. Ranks will allow for certain statements to be marked as preferred or deprecated. This is for example useful to indicate a previous mayor of a city, or the number of inhabitants of a country in 1900.
- Magnus Manske wrote a gadget that allows you to additionally show Wikidata search results when doing a search on Wikipedia. He also extended the Reasonator tool to now also work for cities. Until now, it only supported people and species.
Future
[edit]- The engineering management team continues to update the Deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the annual goals, listing ongoing and future Wikimedia engineering efforts.