Wikipedia Zero/status

Last update on: 2014-11-monthly

2012-02-29

Patrick Reilly continued work on our zero-rated MediaWiki extension. He spent the month cleaning up issues reported by our partners and prepped the extension for more scheduled partner testing in March.

2012-03-15

Patrick Reilly deployed Wikipedia Zero to production in prep for testing with Digi. We're continuing testing through the week

2012-03-21

We continued testing with Digi

2012-03-31

Patrick Reilly deployed the ZeroRatedMobileAccess extension to production, and we tested it with Digi.

2012-04-02

We finished styling issues for the Digi Malaysia launch and are prepping communication

2012-04-monthly

Patrick Reilly and Dan Foy have continued to support our Zero partner roll-outs. We're testing in new countries and are updating the Zero software as necessary. Due to our partners' confidentiality requirements, we haven't been able to highlight the specific areas where testing is happening. Moving forward, we are going to work with our partners to be able to disclose more.

2012-05-02

Patrick has continued to work with Dan about launch related zero issues

2012-05-17

Our focus lately has been working with Orange affiliates in Africa. Right now we are conducting tests in Tunisia, Kenya, Uganda, Cameroon, Niger and Ivory Coast

2012-05-monthly

May saw the first official launch of Wikipedia Zero in Malaysia. Patrick Reilly, Dan Foy and others conducted tests in Tunisia, Kenya, Uganda, Cameroon, Niger and Ivory Coast. Patrick worked with Dan to further improve the Wikipedia Zero extension.

2012-06-monthly

Dan and Patrick continued conducting tests with Orange in six different countries. Additional testing and refinement is under way with our partners in Bangladesh and Montenegro.

2012-07-monthly

Dan and Patrick continued conducting tests with our partners in Bangladesh and Montenegro. We debugged and resolved serious issues with our Opera Mini integration and general infrastructure.

2012-08-monthly

Configuration of partner data is now more configurable and various additional partners are now in testing mode. List of launches to follow.

2012-09-monthly

Configuration of partner data is now more configurable and various additional partners are now in testing mode. List of launches to follow.

2012-10-01

Working on streamlining the IP detection process as well as the overall configuration. New features for partners have also been added that relate to the UI (e.g., custom banner colors).

2012-10-monthly

This month, we've launched Wikipedia Zero with dtac Thailand and STC Saudi. We've also enhanced some diagnostics to inform the users of Zero if they are not using a known partner IP address. We continued to talk with potential new partners and made internal improvements to simplify the configuration process.

2012-12-01

This month we've worked with volunteer developers at the Bangalore DevCamp on our SMS feature, and are preparing for the launch of additional partners in the next few weeks.

2012-11-monthly

This month we've worked with volunteer developers at the Bangalore DevCamp on our SMS feature, and are preparing for the launch of additional partners in the next few weeks.

2012-12-monthly

During the month of December, Patrick Reilly, Dan Foy and the rest of the Zero team launched Wikipedia Zero with a new partner, Orange Congo. They resolved operational issues that prevented the team from accurately recording traffic from the Opera browser. They also helped on-board Brion Vibber to help in the interim while the team continues to look for permanent members. The team is very excited about its upcoming launches and will be announcing them as soon as possible.

2013-01-monthly

During January, Wikimedia was awarded a grant in the Knight News Challenge for our work in expanding Wikimedia mobile projects. Part of this grant will be used for Wikipedia Zero and the SMS/USSD projects to improve access to knowledge in the developing world. In addition, we've partnered with VimpelCom to provide Wikipedia Zero to at least 100 million additional customers this year.

2013-02-monthly

During February, we launched with a new partner, Orange Botswana. We've also begun testing with Vimpelcom for an upcoming launch in March. In addition, we've made improvements to the partner dashboard which tracks Wikipedia Zero usage.

2013-03-monthly

In March we added new telecom partners (such as Axiata Group Berhad) and fixed some bugs, and brought new staff online. We also won an SXSW Interactive "Activism" award for Wikipedia Zero. In April we aim to start improving the code and the IP detection.

2013-04-monthly

We deployed Mobile Web's MobileFrontEnd-ZeroRatedMobileAccess decoupling code to production. We also started the next point release to support more object-friendly JSON-backed carrier preferences, updated carrier preferences, fixed UI button rendering bug, and documented configuration parameters. Last, we added content to wiki pages, and prepared for the migration of non-embargoed content to public wikis.

2013-05-monthly

This month, the team launched Wikipedia Zero with Mobilink in Pakistan, refactored legacy codebase, migrated configuration from monolithic wiki articles to per-carrier JSON configuration blobs, generated utility scripts, patched legacy hyperlink redirect and content rendering bugs, and supported partner on-boarding.

2013-06-monthly

This month, the team launched Wikipedia Zero with Dialog in Sri Lanka, patched logic and user interface bugs, enhanced the configuration editor, expanded logging and debugging for identification of anomalous access, further decoupled ZeroRatedMobileAccess from MobileFrontend, and proposed ESI- and JavaScript-based software re-architecture.

2013-07-monthly

This month, the team launched Wikipedia Zero with Aircel in India, a carrier with about 60 million cellphone subscribers. We also completed our first cut of automation testing, started the implementation of the Wikipedia Zero software re-architecture, and patched bugs. During July we planned for the upcoming year in Wikipedia Zero. On the engineering front we are focusing first on test automation and re-architecture concurrent with SMS/USSD and J2ME releases, and afterward will be focusing efforts on end user UX and carrier-oriented enhancements that will support the continued growth of the program.

2013-08-monthly

This month, the team completed version 1 of Wikipedia Zero automation tests, continued programming the re-architecture of Wikipedia Zero, implemented search engine non-indexing, and analyzed HTTPS requirements in support of a push for greater usage of HTTPS across Wikimedia projects. The Wikipedia Zero engineering team thanks Amit Kapoor from the Wikipedia Zero partnerships team, who wrapped up work with Wikimedia Foundation this month, for his hard work getting the program off the ground. And the team is also pleased to welcome Carolynne Schloeder, who joins the Wikipedia Zero program as Director of Mobile, Programs.

2013-09-monthly

This month, the team released enhanced URL rewriting and debug flag-only Edge Side Includes (ESI) banner inclusion to production, supported the Ops implementation of dynamic MCC/MNC carrier tagging, identified web access log and user agent anomalies, further analyzed and recommended load balancer IP address-related changes in support of HTTPS requirements, and tested JavaScript-based Wikipedia Zero user interface enhancements.

2013-10-monthly

This month, the team released a wide array of features and enhancements. Part of the work focused on reducing accidental data charges, for example by adding interstitials and suppressing UI chrome icons. The latter change caused a 30-minute outage of JavaScript and CSS assets on Wikidata.

We started redirecting ineligible access attempts against subdomains zero.wikipedia.org to a unified warning page; this will reduce residual content in search engine caches, and reduce charges for users accidentally visiting zero.wikipedia.org subdomains while off participating networks.

We added support for partner self-management of configuration for JSON-oriented users; this will be enhanced with stage gating before any actual push to production.

Last, we nearly finalized IP address zero-rating lists (with the support of the Operations team), and we've also started working on Wikipedia Zero support for all Wikimedia projects.

2013-11-01

This month, the team released a wide array of enhancements and supported the BizOps team:

Added support for partner self-management of configuration for JSON-oriented users; this will be enhanced with stage gating before any actual production enablement for specific individuals working for carriers
Added JavaScript and CSS-driven UI chrome suppression of HTTPS-inducing (charge-inducing) UI chrome icons; chrome will be reestablished as each carrier begins supporting IP address-based zero-rating; added modal interstitials to reduce unncessary server roundtrips for most chargeable links; the patch caused a 30 minute outage of JavaScript and CSS on Wikidata (W0 assumes the presence of mobile web, but mobile web wasn't installed on Wikidata while W0 was installed) but was resolved with grace by a number of engineers
Met with carriers and enthusiasts, evangelized the technology, and supported launches
Removed risk of charges on the "No" interstitial links
Enhanced footer links to (almost) universally go through an interstitial where appropriate, instead of direct links inducing charges
Added support for the language support dropdown menu on the Wikipedia Zero landing pages for smartphones; it previously only supported smartphones
Removed risk of showing "free Wikipedia" banners to users visiting File: pages when the carrier doesn't zero-rate File: pages (images, typically)
Established ESI banner testing in the beta labs environment in the march toward production
Started patch to support Wikipedia Zero on all Wikimedia projects
Fixed a reflected XSS bug
Started redirecting ineligible access attempts against subdomains zero.wikipedia.org to a unified warning page; this will reduce residual content in search engine caches (already not visibile without explicit domain name searching), and will reduce charges for users accidentally visiting zero.wikipedia.org subdomains while off-W0 participating networks
Started work on more user-friendly interstitial warning messages
Prepared small patch to constrain transmission of carrier identification headers to the backend for mobile Wikipedias only, to have smaller log files (pageview definitions are unchanged at the moment, though)
Started profiling of some proxy traffic, patched for eventual smarter proxy-aware Wikipedia Zero handling; supported BizOps on technical discussion
Maintained automation test scripts to support WML/WAP tests reflecting newer configuration
Refined the definition of pageviews with Analytics; committed to creating a strawman filtering function for W0-eligible zero-rating based on historical carrier configuration data - this will be one of several filter definitions in the MapReduce jobs
Planned for support of a "global home page" for m.wikipedia.org and zero.wikipedia.org, at least for Wikipedia Zero (if not eventually for all of the mobile web)
Attended the Wikipedia App "reboot" planning meetings; there will be opportunities for Wikipedia Zero if prioritized
Nearly finalized IP address zero-rating lists with Ops' support

2013-11-monthly

During the last month, the team monitored the rollout of Wikipedia Zero via text (USSD/SMS) in partnership with Airtel Kenya and Praekelt for the first pilot of the program. Additionally, Yuri Astrakhan promoted the program abroad.

The team also prepared code and configuration for approval, finalized IP addresses for zero-rating and deployed bugfixes for the Wikipedia app for Firefox OS. We added support for simpler JSON in configuration files, enhanced performance and redirect features and constrained ZeroRatedMobileAccess extension loading to guard against repeats of last month's configuration bug.

2013-12-03

During the last month, the team monitored the rollout of Wikipedia Zero via text (USSD/SMS) in partnership with Airtel Kenya and Praekelt for the first pilot of the program. Additionally, Yuri Astrakhan promoted the program abroad.

The team also

Prepared code and configuration for approval
- Global "mdot" and "zerodot" redirector endpoint
- FlaggedRevs (peer reviewed) JSON configuration files
- Accurate display of Wikipedia Zero experience on proxied (e.g., Opera Mini) connections
Finalized IP addresses for zero-rating (special thanks to Mark Bergsma in Ops on analysis and deployment)
Deployed bugfixes for the Wikipedia app for Firefox OS (special thanks to Max Semenik on deployment), taking future responsibility for this app
Added support for simpler JSON in configuration files
Enhanced performance and redirect features
Constrained ZeroRatedMobileAccess extension loading to guard against repeats of last month's configuration bug

2013-12-monthly

During the last month, the team implemented a global landing page redirector for mobile Wikipedia website access, added support for staged configuration submittals, enhanced interstitials based on input from the field from Wikipedia Zero markets, amended compression proxies support, analyzed USSD/SMS service and partner launch-related access, and added general bugfixes. The team also worked toward a generic JSON configuration extension for use by extension like ZeroRatedMobileAcces, and started on an HTML5 webapp proof of concept as an option for rebooting the Firefox OS app.

2014-01-monthly

During the last month, the team added forward compatibility to Varnish scripting for Wikipedia Zero, and resubmitted a Varnish script patch to support HTTPS for select Wikipedia Zero partners under the new IP address-based zero-rating scheme, after analysis with the Operations team. We also continued proof of concept work on an HTML5 web app for Firefox OS, fixed bugs in the legacy Firefox OS Wikipedia app, and prepared alpha functionality for the integration of Wikipedia Zero with the rebooted Android Wikipedia app. The team also continued work toward a generic JSON configuration extension for use by extensions like ZeroRatedMobileAcces, submitted code for the core MediaWiki API, submitted a ResourceLoader (RL) enhancement and cooperated on alternatives for performance enhancement of RL on non-WMF Redis-backed ResourceLoaders, and submitted a small UX enhancement for the Android rebooted Wikipedia app. January 2014 was also a month of planning: the partners engineering team met for two days with the business development team to plan for partners and Wikipedia Zero-related work at large. The partners engineering team also applied itself to two days of product planning for the Partner Portal. Finally, the team conducted normal tech facilitation to enable partner launches and align approaches with current and future partners.

2014-02-monthly

During the last month, the team added zero-rating for HTTPS for select carriers in cooperation with the Operations team. In collaboration with the Mobile Apps team, we integrated Wikipedia Zero into the forthcoming rebooted versions of the Android and iOS apps, including API and client-side code for zero-rating detection. We updated the legacy Firefox OS app with bugfixes from January (make spinner background opaque, remove mozmarket.js legacy JS); we also prepared other bugfixes for that app (keep last page browsed on low memory crash, avoid text overlaying <select> dropdwon, ensure 'X' clicks stop processing and not send user to Main Page). Discussion with the Operations team and Platform Engineering continued on the ideal portal hosting approach concurrent with sprint planning; portal work is probably deferred until the hosting strategy is formalized. The team also started work on the core API to allow dynamic category pages based on search terms, as well as continuing the discussion on core ResourceLoader features, in support of a proof of concept HTML5 webapp riding atop MobileFrontend. We also started a patch to make contributory features (not just banners and rewritten URLs) present for Wikipedia Zero users on carriers supporting HTTPS zero-rating. Last but not least, Yuri Astrakhan performed extensive analytics work on pageviews and page bandwidth consumption for gzip-capable Wikipedia Zero clients across all Wikipedia Zero-scoped partner pageviews; Yuri also conducted additional analytics work on SMS/USSD data.

2014-03-monthly

During the last month, with the assistance of the Ops and Platform teams, the Wikipedia Zero team added hosting for the forthcoming Partners Portal and continued work on image size reduction for the mobile web. Additionally, the team added Wikipedia Zero detection to the Wikipedia for Firefox OS app and added contributory features support for users on partner networks supporting zero-rated HTTPS connections. The team also removed search thumbnails for zero.wikipedia.org connections in order to avoid spurious charges on devices supporting high-end JavaScript yet using zero.wikipedia.org. Analytics fields were added for the purpose of counting proxy-based and HTTPS-based connections. Routine pre- and post-launch configuration changes were made to support operator zero-rating, and technical assistance was provided to operators and the partner management team to help add zero-rating. The Wikipedia Zero automation testing server was also migrated. The forthcoming Android and iOS apps were also updated to make Wikipedia Zero detection a standard fixture.

Yuri continued analytics work on SMS/USSD pilot data. Post hoc analysis was performed on WML usage after its deprecation; it is still low, although obtaining more low-end phones to check for how well HTML renders and how to enhance the HTML could be useful. Post hoc analysis was also performed on anomalous declines and growth spurts in log lines (not strictly related to pageviews); in the former it much had to do with API changes and in the latter it had much to do with an external polling mechanisms.

With the assistance of the Apps team, User-Agent, Send App Feedback, and Random features were added to the forthcoming reboots of the Android and iOS apps, while making the Share feature for Android allow for a different target app each time and providing code review assistance on the Android and iOS apps code; proof of concept for fulltext search was started on iOS. Wikipedia for Firefox OS bugfixes were also pushed to production. Screencap workflows and preload information was put together for the Android reboot with respect to Wikipedia Zero as well.

The team worked with Ops on forward planning in light of the extremely infrastructure-oriented nature of the program. Quarterly review as held with the ED, VP of Engineering, and the W0 cross-functional team, and the W0 cross-functional team reviewed presentation material for publication. The team also continued work on additional proxy and gateway support. To help partner tech contacts, the team worked on reformatting the tech partner introductory documentation.

Finally, the team explored proactive MCC/MNC-to-IP address drift correction, and will be emailing the community for input soon.

2014-04-monthly

During the last month, the team continued setup tasks on the Partners portal, JSON configuration store, and graceful image quality reduction. The team also updated Android and iOS Wikipedia app reboot visual flourishes for Wikipedia Zero, analyzed anomalous access patterns and proxy-oriented configuration and tech documentation to close gaps, and created bugfixes for unnecessary charge warnings in the "Read in another language" language picker plus direct upload.wikimedia.org image hyperlinks on File: pages. The team also removed some legacy ETL code from the ZeroRatedMobileAccess extension.

Yuri did outreach abroad and continued analytics work on SMS/USSD pilot data. The team also generated two custom pageview analyses for an operator to distinguish traffic by high level device access characteristics as part of ongoing discussions. The team also explored legacy Android Wikipedia app trends.

Additionally, the team cut Android Wikipedia app alpha builds, worked on User-Agent string and URL format updates for the forthcoming iOS Wikipedia app to ensure pageview logging, and performed app code review.

Discussion with the community on MCC-MNC logging to address mobile IP drift was conducted, and it appears it is okay to proceed; the team will reduce the date granularity of log lines to the day (e.g., YYYYMMDD) with a patch to MediaWiki core, though.

Routine pre- and post-launch configuration changes were made to support operator zero-rating, and in-depth technical assistance was provided to operators and the partner management team to help add zero-rating and address anomalies.

The team emailed further about full-text search in reboots of Wikipedia apps, and may resume investigation of it later.

The team also examined requirements for portal and general partners engineering human resources.

2014-05-monthly

During the last month, the team restored IP address dynamic updates for Wikipedia Zero partner configurations, advanced refactoring of ZeroRatedMobileAccess into multiple extensions, added support for graceful image quality reduction (roll-out for Wikipedia Zero will be carefully approached), fixed an HTTPS-to-HTTP redirection bug, and worked on an RFC for GIFification of banners instead of ESI. We also added MCC/MNC sampling and necessary library support to the reboots of the Wikipedia apps, cut an alpha Android build, performed limited app code review, added support for Nokia (now MS Mobile) proxies (next step is to add zero-rating with operators who have a Nokia proxy arrangement), diagnosed configuration retrieval issues, simplified automation tests down to just banner presence checks, ran SMS usage analysis and a one-off operator pageview analysis, and started work with the Design team on the final polish for the Wikipedia Zero experience in the forthcoming apps. Additionally, the team started its annual engineering goal setting for the upcoming fiscal year.

Routine pre- and post-launch configuration changes were made to support operator zero-rating, with routine technical assistance provided to operators and the partner management team to help add zero-rating and address anomalies.

2014-06-monthly

During the last month, the team deployed the refactored Wikipedia Zero codebase that replaces one monolithic extension with multiple extensions. The JsonConfig extension, which allows a wiki-driven JSON configuration system with data validation and a tiered configuration management architecture, had significant enhancements to make it more general for other use cases.

Additionally, the team enabled downsampled thumbnails for a live in-house Wikipedia Zero operator configuration, and finished Wikipedia Zero minimum viable product design and logging polish for the Android and iOS Wikipedia apps. The team also supported the Wikipedia apps development with network connection management enhancements in Android and iOS, with Find in page functionality for Android, and response to Wikipedia for Android Google Play reviews.

The team facilitated discussions on proxy and small screen device optimization, and examined the HTML5 app landscape for the upcoming fiscal year's development roadmap. The team also created documentation for operators for enabling zero-rating with different connection scenarios. Bugfixes were issued for the mobile web Wikipedia Zero and the Wikipedia for Firefox OS app user experience.

Routine pre- and post-launch configuration changes were made to support operator zero-rating, with routine technical assistance provided to operators and the partner management team to help add zero-rating and address anomalies. Finally, the team participated in recruitment for a third Partners engineering teammate.

2014-07-monthly

During the last month, the team worked on software architecture features that allow for expansion of the Wikipedia Zero footprint on partner networks and that get users to content faster with support for lowered cache fragmentation on Varnish caches. Whereas the previous system supported one-size-fits-all configuration for heterogeneous partner networks, inhibiting some zero-rated access, the new system supports multiple configurations for disparate IP addresses and connection profiles per operator. Additionally, lightweight script and GIF-ified Wikipedia Zero banner support has been added and is being tested; in time this should drastically reduce Varnish cache fragmentation, making pages be served faster and reducing Varnish server load. A faster landing page was introduced for "zerodot" (zero.wikipedia.org, legacy text-only experience) landing pages when operators have multiple popular languages in their geography. Work on compression proxy traffic analysis for header enrichment conformance with the official Wikipedia Zero configurations was also performed after more diagnostic logging code was added to the system. Finally, watchlist thumbnails, although low bandwidth, were removed from the zerodot user experience, as was the higher bandwidth MediaViewer feature for zerodot; mdot will have these features, though.

In side project work, the team spent time on API continuation queries, Android IP editing notices, Amazon Kindle and other non-Google Play distribution, and Google Play reviews (now that the Android launch dust has settled, mobile apps product management will be triaging the reviews). In partnerships work, the team met with Mozilla to talk about future plans for the Firefox OS HTML5 app (e.g., repurposing the existing mobile website, but without any feature reduction) and how Wikimedia search might be further integrated into Firefox OS, and also spoke with Canonical about how Wikipedia might be better integrated into the forthcoming Ubuntu Phone OS.

Routine pre- and post-launch configuration changes were made to support operator zero-rating, with routine technical assistance provided to operators and the partner management team to help add zero-rating and address anomalies. The team also continued its search for a third Partners engineering teammate.

2014-09-monthly

During September 2014, the team built more Partners Portal architecture, including Graph extension integration components for eventual display of aggregate statistics to zero-rating partners (it already works and is being reviewed in house). The team also grew support for dynamic zero-rating banners while enhancing JSON configuration extension code and issuing bugfixes. Additionally, the team shrunk the size of the Wikipedia favicon to reduce bandwidth usage by users across the web. And on the partner side, we launched Telenor Myanmar in September.

Finally, the team welcomed its newest software engineer, Jeff Hobson, to the Wikimedia Foundation!

2014-10-monthly

During October 2014, the Partners engineering team enhanced the partners portal (ZeroPortal) extension to improve security and maintainability, added better font support for the Wikipedia Zero end user experience, migrated the remainder of operator partner configurations to use the unified design in support of lowered cache fragmentation on the mobile web Wikipedias, improved the fidelity of zero-rated Graph extension pageview statistics relative to the legacy Limn graphs while streamlining batch processing, made Wikipedia Zero end user information screen overlays follow MobileFrontend style, created Cucumber tests for the partners portal, added colorized crouton support to the Android app while reducing intrusive warnings on the Android and iOS apps for zero-rating (colorized iOS banner code also merged, will be in forthcoming release), prepared analysis and code for language-aware redirects from the m.wikipedia.org webroot, consulted with partners on API usage and app preloads and HTTP header enrichment and language aware redirects, and examined the interplay of Wikipedia Zero with CentralNotice (e.g., fundraising) banners; currently, CentralNotice is suppressed when Wikipedia Zero page rewriting is in force, but in the future it would be nice to show non-fundraising CentralNotice content.

2014-11-monthly

During November 2014, the Partners engineering team continued work on the the partners portal (ZeroPortal) minimum viable product ("MVP"), further improved the fidelity of zero-rated Graph extension pageview statistics relative to the legacy Limn graphs and identified further page filtering criteria, generated statistics on basic pageviews by JavaScript support level in both Wikipedia Zero and general mobile web Wikipedia, added basic scheduled runs of the portal Cucumber tests, added colorized indicator support to the Wikipedia for iOS Wikipedia Zero experience, fixed the broken "x" close button on Wikipedia Zero banners, and finalized the speedier mdot landing page and language-aware redirect code for Wikipedia Zero users. The team also consulted with partners on API usage and zero-rating implementations.