Wikimedia Release Engineering Team/Checkin archive/20151123

2015-11-23

Vacations/Confs/etc

How to do it: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Time_off

• Nov 24: Mukunda - day off (requested in ADP)
• Nov 25: Greg - Half day (working that morning)
• Nov 25: Tyler - Half day (need to update ADP still)
• Nov 25: Mukunda - half day
• Nov 26: US HOLIDAY - Thanksgiving
• Nov 27: US HOLIDAY - Day after Thanksgiving
• Dec 2?-Jan 0x - Antoine, no clue yet
• Dec 4th: Greg - disconnected, leaving Thursday evening, returning Sunday :)
• Dec 2x--XX: Antoine, unscheduled yet
• Dec 21-Jan 1: Greg - vacation, at least (maybe part of the week before)
• Dec 22-29: Chad - Christmas
• Dec 23–25: Tyler — Hopeful, probable, Christmas in Kansas!
• Dec 24-Jan 3: Dan - Holidays
• Dec 25: US HOLIDAY - Christmas Day - December 25
• Jan 1: US HOLIDAY - New Year's Day
• Jan 4 - 8: WikiDev16 + All Hands
• Jan 16-18: Chad - another music festival
• Jan 18: US HOLIDAY - Martin Luther King Day
• Feb 15: US HOLIDAY - President's Day
• May 17-(?): Dan - paternity leave :D
  • PO Box for pastries? - Antoine
• May 30: US HOLIDAY - Memorial Day
• June-ish: Chad - EDC
• August: France holiday - because french. :)

Team Business

Actions from last meeting

• DONE - Greg: Reword the differential dogfooding goal
• DONE - Greg: Get perferred physcian form from HR for US staff
• TODO - Antoine + Mukunda should sit down and talk CI/Harbormaster/Nodepool
  • Mukunda just needs to find time to finally test out harbormaster triggering jenkins jobs.
• IN PROGRESS - Chad: re 1.26 release: "Going to push on all the assigned ones and find assignees for the remainder this week."
  • Dan to continue to bring up in SoS
  • Release is next week, the 25th of Nov. before thanksgiving (US Nov 26th).
• TODO - Antoine: re KPIs, do mw core gate-processing time for this quarter, reassess for next quarter if needed
• TODO - No One Yet: investigate carbon aggregation of stats >1 month old behavior
  • ACTION: Antoine to create a task

WIkiDev16

• Code Review RFC - https://phabricator.wikimedia.org/T114320
• Scap3 - https://phabricator.wikimedia.org/T114045

Other

• Dan: Ask team about ideas for onboarding tasks.
  • https://phabricator.wikimedia.org/tag/engineering_onboarding_template/
  • https://phabricator.wikimedia.org/tag/testing_initiative_2015/ <- add ideas here!

Q3 Goals

https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Goals/201516Q3

Goals timeline:

• December 3: Group goal scoped and drafted on mediawiki.org for Technology team.
• December 10: Group goal + all individual team *drafts completed* on mediawiki.org; discuss at Infra+Tech group and identify dependencies.
• December 17: individual team goals + group goal *finalized* on mediawiki.org; discuss at Monthly Eng Staff.

• Some discussion in scap meeting Friday

• scap3 for mediawiki?

• Beta Cluster Cleanup (questionable)
  • Incorporate the work from the previous Staging effort into the current Beta Cluster
    • specific outputs?

• Differential
  • Garner more early adopters (people and projects)
    • how many?
    • one proj per team (at WMF) in differential
      • reading
      • editing
      • discovery
      • "technology"?
        • operations repos that don't have much CI (eg: deb packages)
        • research? performance?

• Reduce CI wait time
  • we still need a graph for the wait time that's clear :/
  • https://grafana.wikimedia.org/dashboard/db/releng-zuul

KPIs

https://grafana.wikimedia.org/dashboard/db/releng-kpis

• Log spam:
  • Can we explain the two bumps (up and down) of the INFO lines?
    • RunJobs logging changes? See engineering-l/ops-l? (greg was lazy this morning, will look later)
  • Looking good, actually. The log(n) hides it a bit
    • on 9/12 we were at 1k warning, 2k errors
    • last week we've been between 100-800ish for both
    • we should make some graph that shows that more clearly
    • ACTION: Chad take a look at making the >=Warning graph (mostly done, trying to figure out right log)

New vs Maint time spent

• https://docs.google.com/a/wikimedia.org/spreadsheets/d/1FI90AefwdLHGzVVdrLS6AxcTcJtLFyX0aQTQfyc88s4/edit

#together

• Team workboard: to triage: https://phabricator.wikimedia.org/project/board/20/query/TRiVy4zOMdR./
• Team workboard: only-in-#releng(ish): https://phabricator.wikimedia.org/project/sprint/board/20/query/g2T5.QSLJVRQ/

Scrum of Scrums

https://phabricator.wikimedia.org/project/board/64/

Blocked on us: https://phabricator.wikimedia.org/maniphest/query/h7YTCBTJsepS/#R

Project Updates

CI Scaling

https://phabricator.wikimedia.org/project/board/1010/

Quarterly Goal: "CI cluster responds to spike in queued builds by starting and registering additional jenkins slaves" - https://phabricator.wikimedia.org/T111106

• zuul-cloner being added
• instances now have git mirror of mediawiki/core and operations/puppet. Will add more depending on size / jobs frequency.
• Have jobs to clone source code and work from /srv/ .
• Then get a convention of capturing artifacts in /log/** /build/** .

Beta Cluster

https://phabricator.wikimedia.org/project/board/497/?order=priority

• Antoine / Jaime chatted about databases performances. Random items:
  • https://phabricator.wikimedia.org/T116793 Investigate slow query logging/digest for Beta Cluster
  • https://phabricator.wikimedia.org/T108255 Enable MariaDB/MySQL's Strict Mode (Antione and Jaime)
  • Migrate to Jessie / MariaDB > 5.5 (currently precise) (with Jaime)
  • Enable performance_schema (just antoine?)
  • Hook db1/db2 in tendril
  • Budget for real hardware, ie have beta cluster DB on labdbXXXX hosts

=> Antoine to summarize discussion with Jaime and fill tasks accordingly

• Add them to https://tendril.wikimedia.org

• Team: might have to watch slow query logs / strict errors etc

Deployment Cabal

Main: https://phabricator.wikimedia.org/project/board/349/

Scap3: https://phabricator.wikimedia.org/project/board/1449/

Quarterly Goal: "Migrate all Service team owned services and MW deploys to scap3" - https://phabricator.wikimedia.org/T109926

• scap 3.0?
  • no idea / pending production use

Diff[usion|erential] migration

https://etherpad.wikimedia.org/p/diffuerential-weekly

Gitblit-Deprecate: https://phabricator.wikimedia.org/project/board/46/

Quarterly Goal: https://phabricator.wikimedia.org/T111465

Gerrit-Migration: https://phabricator.wikimedia.org/project/board/9/

• Find some Differential Ambassadors for each vertical
  • bd808 for Reading
• Antoine has arcanist installed!  ;)

• gitblit redirect stuffs
  • chad wants more out of mukunda :)
  • merge the first part (today), still work on the extra bits

Release MediaWiki 1.26

https://phabricator.wikimedia.org/project/board/1177/

Release date 11/25

• Do we have all bundled extensions/skins migrated to extension.json/skin.json?
  • MediaWiki 1.26 bundled repo should be state of the art https://phabricator.wikimedia.org/T115392
  • confirmedit is ready for backport

Other Work

• Mukunda is working on subtree merging mediawiki extensions for long-lived release branch.