Scrum of scrums/2015-08-05

2015-08-05

Reading

Android app

• Published beta 107 on Monday. Published beta 108 on Tuesday to add some bug fixes.
• Sent off bits to TSG. If all goes well we will publish a stable release sometime next week. https://phabricator.wikimedia.org/T107344
• Updated folder structure in our Android app repo to follow Gradle standard structure. Best to reclone repo and reimport into Android Studio.

Discovery

• Identified sources of several mass zero-result queries, some are internal (we have met the enemy and they are us), fixed since
• Nice dashboards with KPIs: http://searchdata.wmflabs.org/metrics/
• WDQS service running on prod hardware, hooking to varnish/query.wikidata.org next
  • Still waiting for that security review
  • Preparing presentation for http://smartdata2015.dataversity.net/
• Maps Blockers:
  • Add REDIS to maps cluster (ops) - https://phabricator.wikimedia.org/T107813
  • Maps security review (sec) - https://phabricator.wikimedia.org/T105051
  • Pupetize tilerator for depl (ops) - https://phabricator.wikimedia.org/T105074
• Blocking:
  • Nothing on board: https://phabricator.wikimedia.org/tag/blocked-on-search/

Community Tech

• Published Scope and Roadmap (https://www.mediawiki.org/wiki/Community_Tech_team )
• Working on simple syntax highlighter for Wikitext editor
• No blockers

Editing

Language

• Blocking: None'
• Blocked: None
• Updates:
  • ContentTranslation reached 13K mark; Lots of publishing related issues are fixed in next deployment.
  • Kartik is updating Apertium* packages as part of Jessie migration (and as service runner migration of cxserver)
  • https://phabricator.wikimedia.org/T107270 is worth mentioning, working with Apertium upstream on it.

Parsing

• Dedicated focus since last week towards addressing <nowiki> complaints in VE edits
• Ongoing cleanup of parser tests to get Parsoid and PHP parser output more in sync
• Work in progress to batch Parsoid's request to the mediawiki API ( https://gerrit.wikimedia.org/r/#/c/226670/ and https://gerrit.wikimedia.org/r/#/c/227208/ )
• Starting work to replace Tidy with an alternative - https://phabricator.wikimedia.org/T89331

Collaboration

• We've had some product roadmap changes for the next two quarters, so we're not currently working on search or the blockers for it.
• We will still work on some OOjs UI conversion, but not the whole thing, since that was mainly for search.
• We enabled Flow on almost all wikis. We left a small number disabled initially (private and special wikis like loginwiki), then disabled a few more due to namespace name conflicts. It is not actually on use on the vast majority of these new wikis, but if you see new Flow issues, let us know.
• We're also working on some technical debt issues, such as using content models in a more standard way and fixing a long-standing DB issue.
• We are in communication with ops (particularly Jaime Crespo) regarding https://phabricator.wikimedia.org/T106386 / https://phabricator.wikimedia.org/T106363 / https://phabricator.wikimedia.org/T107610 . Our understanding is Flow is not blocking the short-term solution (hardware purchase) to External Store, and we have an initial plan for the medium-term (moving Flow to its own logical External Store so the regular External Store can be compressed).

Fundraising Tech

• Deployed PCI (SAQ-A)-compliant redis-backed message queue to payments
• CentralNotice refactor merged at PS46! (to feature branch): https://gerrit.wikimedia.org/r/221759/
  • proceeding with more browser tests and mixins to replace wiki-edited JS
• Reviewing plan for Amazon payments switchover
• more DonationInterface fixes and small CRM features

Infrastructure

Analytics

• Data loss in kafka due to an upgrade gone bad. We lost data for all webrequest topics, the "parsoid" topic, and "statsv". Approximate time: 2015-08-03 17:21 UTC to 2015-08-03 23:57 UTC. Some data present, just unreliable due to random loss. Question for y'all: do you want more information on this, were you running important experiments and need to know the level of loss?
• Scalable Event System - had a use case discussion, will talk about technical details next, let us know if you want to be involved
• Pageview API dev in progress

Services

• RESTBase
  • all 9 servers are now up and running!
  • SQLite back-end as a drop-in replacement for Cassandra
    • should land in MW-Vagrant shortly
  • improvements, fixes, etc.
• talking to research and analytics about revscoring and pageview APIs

Security

• https://wikitech.wikimedia.org/wiki/Incident_documentation/20150729-HttpsRedirects
• 1.25.2 should be deployed this week
• Reviews done for: climate, Ex:PageBanner, firebase/php-jwt, composer/semver; Maps this week

Release Engineering

• Blocking: (none)
• Blocked by: operations
  • https://phabricator.wikimedia.org/T72181
• Updates:
  • Nutcracker issues in Beta Cluster persist somewhat
    • https://phabricator.wikimedia.org/T107538
  • Deployment tooling work continues
    • Experimenting and evaluating an Ansible based tool called scale (https://github.com/thcipriani/scale)
  • Voting mwext-mw-selenium job for MobileFrontend, working reliably so far (but slow; 7-12 minutes for 16 scenarios)
    • Investigating ways to speed up browser test execution (parallel execution or simply optimizing framework and common steps)

Research

• Moving ORES "revscoring" (python, distributed processing cluster & machine learning) toward production.
• Working towards "scored revisions" beta feature. FYI: Services Team and Operations are looped in.
  • To-do: Talk to LegoKTM about implementation of beta feature. I don't think he's planning to use RESTBase
• Article recommendation API still working in isolation.