Wikimedia Platform Engineering/MediaWiki Core Team/Check-ins/20140113

Who: Brad, Chad, RobLa, Aaron, Chris, Nik, Bryan, Ori, Antoine

Need post-mortem for CategoryTree related downtime earlier today. Greg to followup with Sam

Bug 59957: Gerrit host has weird CPU usage Gerrit outage over the weekend. Chad will investigate + update bug.

lsearchd hasn’t updated in a week (bug 59979)
- Chad and Nik have looked, somewhat stumped.
- Going to research further, will prod Ori/Tim/ops as needed
- We’re poking Leslie on IRC now
Cirrus now secondary search backend on enwiki
enwiki job queue backed up (not our fault, but we noticed)
- 1.7mil htmlCacheUpdate jobs and rising (+30k since I started watching it this morning)
- Raising job runners by 1 per box: https://gerrit.wikimedia.org/r/#/c/107189/
  - Seems to have helped, numbers going down again (-45k, roughly?)

Aggregated and plotted page load timing data for last 10 months: <http://noc.wikimedia.org/~ori/metrics/page-load.html>.
TODO: Automate the analysis and produce a weekly report to engineering@ / wikitech-l.
Presented at January Metrics Meeting:
Lots of UniversalLanguageSelector analysis and back-and-forths prompted by bug 59958. Reviewed architecture in bug 56433#c11, culminating in Erik’s decision in 56433#c22 to disable automatic font loading, which is a good outcome.
Tracked down core test failure under HHVM in bug 55532#c2. Good news: fixed upstream.
Tim’s xhprof patch merged to HHVM: https://github.com/facebook/hhvm/pull/1488
Andrew Otto ready to configure profiling stats endpoint on bits, waiting on me to e-mail ops@ to get a nod of approval from Faidon or Mark.
Still thinking about performance guidelines for developers.

Extensions on beta no more self deploying bug 49847
- mediawiki/extensions.git no more updated by Gerrit. Need to write some script to make it happen for us.

~80 applications received so far.
No serious bugs found but several small changes to text and validation have been requested and deployed.