Wikimedia Release Engineering Team/Deployment pipeline/2019-03-28
Last Time
[edit]Next Quarter Goals
[edit]General
[edit]TODOs from last time
[edit]
In progress TODO various attack vectors document to start
- antoine and I started to talk about it
- thcipriani to more thoroughly noodle
- In progress TODO: support documention like the one tyler did for the portal and pipeline/helmfile and deployment
- https://wikitech.wikimedia.org/wiki/Deployment_pipeline now exists, https://wikitech.wikimedia.org/wiki/Continuous_Delivery has been deleted.
- reached out to technical writer, in very preliminary stages for next quarter
- think about other documentation needs beyone Deployment_pipeline main page
- TODO add to RelEng offsite list
- TODO: Joe & James_F to work on eventual 2019-04-01 email
- Beware: announces on 04/01 can be considered an April's fool [and James is out on 1 April itself. <-- this is an april fool's joke for sure]
- postponed until there are docs ~2019-05-xx
- Draft: https://etherpad.wikimedia.org/p/LZKuNSFpM4iKbuHkdPRf
RelEng
[edit]- Additional NPM packages during container image build
- Left suggestions for alternative approaches
- Implementing `.pipeline/config.yaml`
- A basic execution graph (directed acyclic graph thingy) is done. Need to work on mapping it to Jenkins Pipeline stage definitions.
- Is Form 3 in the Graph examples coherent?
- Perhaps we don't need separate pipelines defined in `.pipeline/config.yaml` if we have DAG execution (allowing pipelines from multiple projects in the same repo to run parallel or intersect)
- Jenkins has limitations on parallel execution but same graph may map to true DAG execution in [unknown future CI system]
- A basic execution graph (directed acyclic graph thingy) is done. Need to work on mapping it to Jenkins Pipeline stage definitions.
- Integration of dependent changes pre-merge
- james: we currently run into this situation (A and B both have changes which mutually depend on each other) and we have a mess
- We tell people to build forwards/backwards compatibility in both A and B and then merge but (a) that's expensive of developer time, and (b) potentially fragile/cache expensive/etc.
- Sometimes the circular dependencies are only sanely fixable with forced-merges (which we don't want to encourage, or even allow?), which is bad.
- The alternative (supporting circular dependencies and merging them together) means we'd then need a system to keep track of which versions of A and B are each compatible, which could also be a mess.
- dan: the zuul merger creates possible futures via the dependent pipeline currently, but we don't use it that way
- it currently applies to git futures, but it could be leveraged to create deployment futures? (seems rather complicated)
- james: we currently run into this situation (A and B both have changes which mutually depend on each other) and we have a mess
- CI Future WG report: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/CI_Futures_WG/Report
- Feedback welcome
- tl;dr: "The working group recommends that RelEng looks deeper into GitLab, Argo, and Zuul v3, and implements a prototype self-serve CI system, integrated with Gerrit on at least one of them." (but plz read)
Serviceops
[edit]- effie finished cxserver switching
Done
Services
[edit]- TODO merge/deploy changeprop patch for zuul https://gerrit.wikimedia.org/r/#/c/integration/config/+/496387/