Wikimedia Release Engineering Team/SSD Sync Up/2019-07-16

2019-07-16

Last Time: 2019-07-02 Goals: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2019-20_Q1#Release_Engineering

Deployment Pipeline

Workboard

• Last week's Tech Talk by Alexandros: https://www.youtube.com/watch?v=i0FTcG7PxzI

GOALS

• Migrate restrouter
  • https://phabricator.wikimedia.org/T223953
  • (Stretch): MobileContentService
  • (Stretch): Preparatory MediaWiki config clean-up & static loading work 
    • Moving configuration to static configuration (e.g., json files rather than a bunch of PHP)

TODOs from Last Time

•  Done thcipriani -- Pipeline image build cleanup for contint1001 (not that it's important anymore :))
•  Done contint1001 store docker images on separate partition or disk
• In progress thcipriani -- Base Blubber policy file for CI
  • blocked on deployment
  • TODO thcipriani to ping alexandros
• In progress brennen -- pipeline config validation
• N Not done TODO Pipeline docs

New CI

GOALS

• POCs of GitLab and Zuul3 systems; evaluate options
• Document an implementable architecture for what we want in new CI

TODOs from Last Time

• TODO Talk to SRE about Zuul v3 needs
  • Needed: conversation about phasing out Zuulv2
    • Meeting for zuulv2 thursday, TODO ping greg about antoine going
  • Lots of ansible to setup Zuulv3
  • 'TODO Discuss with SRE Thursday

Other Work

• v2 of CI arch doc: https://docs.google.com/document/d/1EQuInEV-eY_5kxOZ8E1qEdLr8fb6ihwOD9V_tpVFWuU/edit
  • Only a few comments received, no significant changes. v3 someday, but not immediately.
  • No news, but s'ok

• Lars is hacking up new components around GitLab (mostly independent of what CI engine we choose, GitLab is just the first one to be tried)
  • going slowly this week, due to train
  • All but one component done (the controller)
  • other components tested manually
  • prototype: http://git.liw.fi/wmf-ci-arch/tree/
    • gitlab-ci uploads build artifacts to the artifact store
    • Testing with http://git.liw.fi/heippa/tree/.gitlab-ci.yml -- simulates the commit stage
    • uses the ick artifact store
    • VCS Worker and Deployment Worker in api.py
    • Deployment via ansible currently (git clone)
    • Aim: get controller working, then get a volunteer -- i.e., build blubber for a prototype

Local Development

GOALS

• Migrate local-charts to deployment-charts
  • https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/522584
    • Did not handle any concurrency with 30 requests/sec
    • Didn't use too many resources
    • Blocked on: Brennen working on moving mw-base image to apache
• Instantiate testing and linting of helm charts
• Preliminary work on a CLI for setup/management

TODOs from Last Time

• mediawiki/core blubber: Jam a shell script into the builder?
  • TODO: the actual jamming of shell script
    • Haven't been able to make that work
    • Job to run docker-pkg nightly to make up-to-date dev image
    • thcipriani/brennen pair Friday on working on this

Other Work

• In progress Porting from local-charts to deployment-charts ( https://phabricator.wikimedia.org/T224935 ):
  • Add restbase chart (port from local-charts) TODO Thursday meeting
•   In progress Move dev-images PHP image from php -S to Apache + php-fpm
  • See also https://gerrit.wikimedia.org/r/c/integration/quibble/+/516729/