Wikimedia Release Engineering Team/Deployment pipeline/2020-11-24

2020-11-24

Always

• [1]
• Archive
• Task tree: https://phabricator.wikimedia.org/T198901

TODOs from last time

• Thcipriani: email fandom re #mw-on-k8s + ask for presentation
  • Done you all should have it as of 2020-10-14

General

• How many meetings do we have about this? (for some value of "this")?
  • Manager sync meeting
  • RelEng Dev meeting
  • RelEng Team meeting
  • Dev meeting
  • This meeting
• TODO: thcipriani cancel these(deployment-pipeline meetings), dduvall fold other dev meeting into this one

RelEng

• Shellbox open question: https://phabricator.wikimedia.org/T263295
• Coordination re:Flink?
  • https://phabricator.wikimedia.org/T264006
  • Which repo to run the pipeline on -- blubber works
  • Have not run scaffold.sh yet
    • combining scaffold with apache docs -- patchset may be incoming there
  • AntiPattern: replicating upstream dockerfiles with blubber
  • Created super-repo: https://gerrit.wikimedia.org/g/wikidata/query/flink-rdf-streaming-updater/+/refs/heads/master
  • Orchestration framework: flink; bundled with application, on top of k8s
  • Costly on merge: pulls flink from the internet and verifies with GPG
  • Potential race condition: change merged, CI job to publish to archiva + blubber job that uses archiva
    • Trigger currently is archiva job is manual, also want blubber job manual
  • TODO: more clarification needed

Serviceops

• Kubernetes staging cluster slowly being setup in codfw with all new version components (calico 3.17, kubernetes 1.16)
  • Revisiting a lot of assumptions and questions for setting up a kubernetes cluster again
  • Impacts CI since CI talks to staging for helm test
• service-checker support for not verifying TLS certs (should help with kask integration tests) sent out, waiting for reviews

As Always

• Release Pipeline Workboard
• Meeting notes