Jump to content

Wikimedia Engineering/Report/2011/November

From mediawiki.org
October 2011 Wikimedia engineering report, November 2011 December 2011

Major news in November include:

  • The completion of the Coding challenge, and two coding events in India and the UK;
  • Continued infrastructure work in our data centers to improve performance and reliability, and on the Labs project;
  • Progress on the Visual editor and its back-end;
  • New versions of the Feedback Dashboard and the Upload Wizard, bringing new and long-awaited features;
  • Fundraising engineering going full-swing, in parallel with the annual fundraising campaign;
  • The final release of MediaWiki 1.18.0.

Hover your mouse over the green question marks ([?]) to see the description of a particular project.

Events

[edit]

Recent events

[edit]
  • India hackathon (18–20 November, Mumbai, India) — Approximately 80 Indian participants came to this week-end hackathon, focusing on Wikimedia mobile, offline, and language issues. Participants made great strides in mobile. The internationalization effort also benefited, with new input methods for MediaWiki, readying Narayam for Wikimedia Incubator, a prototype onscreen keyboard built in Narayam, Wikimedia Mobile ready for translation, new UI prototypes for language selection, and other new features. Participants localized Kiwix into several Indic languages. Volunteer Development Coordinator Sumana Harihareswara summarized the event on wikitech-l.
  • Brighton hackathon (19–20 November, Brighton, England) — Lewis Cawte and Tom Morris organized a general MediaWiki hackathon with approximately 10 participants. Attendees fixed bugs, reviewed outstanding patches from volunteers, enhanced OpenPlaques with the MediaWiki API, and discussed Wikinews and Wikimedia Commons.

Upcoming events

[edit]
  • San Francisco hackathon (21–22 January 2012) — Erik Möller and Sumana Harihareswara continued to plan and started publicizing an outreach-focused developers' week-end. Experienced staff and volunteer developers will participate, teaching new developers about MediaWiki, the API and our framework for JavaScript feature development.

Personnel

[edit]

Job openings

[edit]

Are you looking to work for Wikimedia? We have a lot of hiring coming up, and we really love talking to active community members about these roles.

  • Requests for proposals:
    • API Logging Analysis — Help us analyze the query logs of our API to better understand third party application usage of Wikimedia content and services.
    • XML Dumps — Help us improve the infrastructure used to build XML dumps of Wikipedia content, for backups and reuse by third parties.
    • Mobile UX — Help us redesign our mobile platform and apps as more and more visitors access Wikipedia and its sister sites via mobile devices.

Short news

[edit]

Operations

[edit]

Site infrastructure

[edit]
  • Data Centers [?] — Eight new database servers for external storage were deployed in Ashburn and Tampa, to retire about 30 aging servers. The goal was to add capacity, improve performance, consume less power and provide cross data-center data recovery and redundancy (SDTPA & EQIAD). This gave us back one rack of server space, which is significant given the space constraint at SDTPA. Also in EQIAD, servers for bits.wikimedia.org and Fundraising reporting were migrated and are now in production. In Amsterdam, SSDs were added to the ESAMS Squid servers to improve read performance, and we upgraded the ESAMS core switch (csw2-esams). The KNAMS router was moved because we changed to a new hosting provider, though still within the same datacenter. In Tampa, 66 servers were retired, donated and shipped to various non-profits. More generally, our configuration management tool Puppet was upgraded and deployed to all of our lab and production servers; a Puppet dashboard was also implemented.
  • New MySQL Package — A new Debian package for mysqlatfacebook 5.1.53 was successfully built, using the fresh version of the facebook patchset. Initial production data on two hosts look promising.

Testing environment

[edit]
  • Virtualization test cluster [?] — OpenStack Nova was upgraded from cactus to diablo. A GlusterFS filesystem was added on all compute nodes via puppet, to act as storage for the instances. A default sudo policy was also added for instances: project members now have sudo permissions, excluding global projects. Shared home directories are also available in a per project manner. 15 projects and 36 instances have been created and 46 people have been given Labs accounts so far. The GlusterFS installation and the recent Puppet master and client upgrades were implemented and tested in Labs before going into production.

Backups and data archives

[edit]
  • Data Dumps [?] — While the dumps keep rolling along, we are talking with another organization interested in mirroring them. If you know someone with several terabytes of space who might be interested, please send them our way. All hosts had their kernel updated for security reasons, and new dump code was deployed as well. We rolled out a new experimental service this month of daily adds/changes dumps for all projects. No information about deleted/undeleted/moved pages from previous dumps is included, but it does include all new content since the run of the previous day. The first adds/changes run for the English language Wikipedia took less than 30 minutes to build.

Features Engineering

[edit]

Editing tools

[edit]
  • Visual editor — Trevor Parscal fixed issues blocking the synchronization of structural edits to the user interface, refactored and cleaned up the code, and mapped out tasks and features to be supported. He also finished the document transaction functionality and made progress on an undo/redo system. Roan Kattouw added tests, rewrote some code to make the tests pass, and fixed a number of bugs and issues, notably in Internet Explorer. Inez Korczynski continued to work on content insertion, deletion and selection and fixed numerous bugs. Gabriel Wicke extended the PEG parser for robust larger-scale parsing. He converted the PEG parser into a combined wiki and HTML tokenizer that feeds to a HTML5 DOM tree builder. He implemented several wikitext features (lists, italics, bold) as token stream transformations. 139 of about 660 parser tests are now passing.
  • Internationalization and localization tools — The code of the WebFonts extension was reviewed, and deployment is planned for December. The Narayam input methods extension now remembers the five last used schemes, and a basic version of message group workflows was implemented in the Translate extension. A major focus of the team in November was the India hackathon, during which a lot of work got done. New Narayam keyboard mappings were developed, and WebFonts were tested further.

Participation and editor retention

[edit]

Multimedia Tools

[edit]
  • UploadWizard — Neil Kandalgaonkar and Ian Baker deployed a set of important improvements, including multi-file selection for browsers which support it, custom wikitext licenses, an improved licensing workflow, basic support for location data extraction, and more. Support for chunked uploading (which improves reliability of large file transfers) was temporarily backed out and is still being worked on.

MediaWiki infrastructure

[edit]
  • ResourceLoader — Roan Kattouw wrote a maintenance script for migrating gadgets on a wiki from the old format to the new Gadgets 2.0 format. Roan and Timo Tijhof also fixed several bugs.

Media Labs

[edit]

Mobile

[edit]
  • Mobile Research [?] — Mani Pande and Parul Vora presented the findings from India and Brazil report to the mobile team. They are currently working on consolidating the report, videos, photographs and other media into a Wiki format. The cleaning of data from the mobile survey continues.
  • Android Wikipedia App [?] — We're tidying up the app now and would love to get testers. Send a mail to the mobile-l-feedback list if you want to be involved.

Special projects

[edit]

Fundraising support

[edit]
  • 2011 Fundraiser — The annual fundraiser launched, with support for 76 new credit card currencies, including some which have been long-desired: Rupees, Russian Rubles, and Brazilian Reals. The fundraising engineering team added support to the DonationInterface extension for JCB credit card donations, BPay in Australia, 3 new real time banking options including iDeal, direct debit for 6 countries, manual bank transfers in more than 50 countries, and Webmoney, with more on the way. DonationInterface also benefited from enhancements to the RapidHtml form templating system. From an operations perspective, databases were migrated for increased capacity and stability; the data center failover capacity was also consolidated by adding aluminium.wikimedia.org as a mirror of grosley.wikimedia.org (different data centers, identical hosts), which provide a bulk of our miscellaneous fundraising services (including CiviCRM). We also added the ability to have translated 'thank you' emails to our ThankYou module for Drupal/CiviCRM. The ContributionReporting extension was enhanced to allow custom selection of fundraising years to display, but the feature was disabled after it caused a site outage due to cache stampeding. The FundraiserLandingPage extension was developed and deployed, making it easier to dynamically construct template calls for fundraiser landing pages depending on a potential donor's country. Last, the team fixed a number of new bugs and issues, surfaced by the increased usage of the donation pipeline.

Offline

[edit]
  • Kiwix UX initiative [?] — Version 0.9 beta4 was released in November. Emmanuel Engelhart implemented a new filtering/sorting system in Kiwix's content manager. A new tool, kiwix-install, now makes it trivial to create new portable versions of Kiwix, with the ZIM file and full text search included. kiwix-install also made it possible to implement automated software and content updates. At the India hackathon, localisation for 3 new Indic languages was added, and the Kiwix team met with several organizations to discuss partnerships. Work has started on a Kiwix app for Android/ARM.

Platform Engineering

[edit]

MediaWiki Core

[edit]

Wikimedia analytics

[edit]

Technical Liaison; Developer Relations

[edit]
The "Technical Liaison; Developer Relations" team was featured on the Wikimedia Tech blog this month.

Future

[edit]

The engineering management team continues to update the Software deployments page weekly, providing up-to-date information on the upcoming deployments to Wikimedia sites, as well as the engineering roadmap, listing ongoing and future Wikimedia engineering efforts.

Retrieved from "https://www.mediawiki.org/w/index.php?title=Wikimedia_Engineering/Report/2011/November&oldid=6584870"