Wikimedia Release Engineering Team/SSD Sync Up/2019-07-23

2019-07-23

Last Time: 2019-07-16 Goals: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2019-20_Q1#Release_Engineering

Deployment Pipeline

Workboard

GOALS

• Migrate restrouter
  • https://phabricator.wikimedia.org/T223953
  • Chart published
• (Stretch): MobileContentService
• (Stretch): Preparatory MediaWiki config clean-up & static loading work 
  • Moving configuration to static configuration (e.g., json files rather than a bunch of PHP)

TODOs from Last Time

• In progress thcipriani -- Base Blubber policy file for CI
  • blocked on deployment
• TODO fix trigger jobs aren't concurrent
  • concurrent; true
• stalled brennen -- pipeline config validation
  • TODO brennen to calendar stalk pairing with dan
• In progress TODO Pipeline docs
  • https://wikitech.wikimedia.org/wiki/User:Thcipriani/Deployment_pipeline

New CI

GOALS

• POCs of GitLab and Zuul3 systems; evaluate options
• Document an implementable architecture for what we want in new CI

TODOs from Last Time

• In progress TODO Talk to SRE about Zuul v3 needs
  • see doc: Wikimedia Continuous Integration Infrastructure: Near Term

Other Work

• v2 of CI arch doc: https://docs.google.com/document/d/1EQuInEV-eY_5kxOZ8E1qEdLr8fb6ihwOD9V_tpVFWuU/edit

• GitLab PoC
  • gerrit stream-events
  • http requests to controller

• Argo PoC
  • Gerrit webhooks
  • Argo events
    • Listens for external events
    • sends to "sensor" to create kubernetes object
    • kubernetes argo workflow
  • workflow -> update gerrit patchset
  • minio for artifacts (s3-compat service)

• Webhooks vs stream-events
  • webhooks: per-project
  • gerrit kafka stream-events storage
    • https://gerrit.googlesource.com/plugins/kafka-events/
  • https://gerrit.wikimedia.org/r/plugins/hooks/Documentation/index.html

Local Development

GOALS

• Migrate local-charts to deployment-charts
  • https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/522584
    • In progress Brennen working on moving mw-base image to apache
• Instantiate testing and linting of helm charts
• Preliminary work on a CLI for setup/management

TODOs from Last Time

• mediawiki/core blubber
  • thcipriani/brennen paired https://integration.wikimedia.org/ci/job/mediawiki-pipeline-dev/
  • missing .pipeline/config.yaml/.pipeline/blubber.yaml
    • TODO add branch filter for this job to only trigger on master

Other Work

• In progress Porting from local-charts to deployment-charts ( https://phabricator.wikimedia.org/T224935 ):
  • Add restbase chart (port from local-charts)
•   In progress Move dev-images PHP image from php -S to Apache + php-fpm
  • See also https://gerrit.wikimedia.org/r/c/integration/quibble/+/516729/