Jump to content

Wikimedia Release Engineering Team/Deployment pipeline/20170725-planning

From mediawiki.org

2017-07-25

[edit]

Existing meetings

  • Phabricator task hierarchy: https://phabricator.wikimedia.org/T170453This meeting will be monthly

Purpose of this meeting: Longer term. What is needed during the upcoming month. 

Team-by-team status

[edit]

Ops

[edit]
  • WIthin last couple quarters
    • Fixed Puppetization
    • Created production clusters
    • Created staging cluster
    • Worked a bit on base images and how to manage docker security updates in production (ideas)
  • This quarter
    • Kubernetes: Try to upgrade to 1.7; trying to keep 2 clusters in sync
    • Alex working on network isolation and pods
    • Try to define standard structure (container plus helper containers)
  • Current/upcoming dependencies
    • Not at the moment. Will work with cloud on Kubernetes upgrade. 
    • Will want feedback after the plan is written

Services

[edit]
  • Created basic minikube test cluster: https://github.com/wikimedia/mediawiki-containers/blob/master/README.k8s.md
    • Started work on `mwctl` utility to manage edit -> test workflow: https://github.com/wikimedia/mwctl
    • Design notes at https://docs.google.com/document/d/1Z1nPkWea-YtWoYAvHVK8SNoD7gM8yRGf_VbiRkWalsw/edit#
  • RFC on container structure standards: https://phabricator.wikimedia.org/T169998
  • Marko working on documenting use cases
  • Next step: Make it a product
    • mwctl tool: allow you to test and develop code in minikube environment
    • RFC - https://phabricator.wikimedia.org/T169998 (see also above)
      • Status of RFC: Parts don't matter, but waiting on us (in this call) to agree on them
      • mwctl relies on conventions. So ideally resolve the RFC "soon" (the sooner the better)
  • Blocked/dependencies
    • Document use cases (Marko will write and circulate)

Release Engineering

[edit]
  • Worked on abstraction to generate images for various stages of pipeline (dev/staging/prod)
    • https://phabricator.wikimedia.org/source/blubber/
  • review of acl for jenkins for a where to build these containers for pushing into staging pipeline
    • https://phabricator.wikimedia.org/T169557 (could use more review)
  • Dan starting review of deployment/k8s packaging stuff with helm -- to sync up with services work
    • relates to mwctl work
  • Dependencies
    • acl is the big thing (we have discussed at weekly meetings)
    • Need a bit more sync-up with ops about orchestration tools (e.g. helm)...will be blocked early next quarter

Next month, this meeting should start to think about Q2 goals for the program

Program status

[edit]

https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2017-2018/Final/Programs/Technology#Program_6._Streamlined_service_delivery Outcomes, Objectives, and Milestones Outcome 1: We have seamless productization and operation of (micro)services.

  • Objective 1: Set up production-ready Kubernetes cluster(s) with adequate capacity
    • Done (can expand capcity as needed with budget)
  • Objective 2: Create a standardized application environment for running applications in Kubernetes
    • Working on it this quarter (and next)

Outcome 2: Developers are able to develop and test their applications through a unified pipeline towards production deployment.

  • Objective 1: Create guidelines and abstractions for building and testing applications in containers
    • Work ongoing by Ops (but not tied to a quarterly goal)
  • Objective 2: Set up a continuous integration and deployment pipeline to publish new versions of an application to production via testing and staging environments that reliably reproduce production
    • Quarterly goal by RelEng
    • Includes Blubber
    • Define functional tests for Mathoid running on the staging Kubernetes cluster for use in future gating decisions - task T170482
    • Define method for monitoring and reacting to the above functional tests - task T170483
  • Objective 3: Provide a lightweight integrated development environment that lets developers test their code against a local miniature copy of the production stack
    • Work ongoing by Services; prototype ready. (but not tied to a quarterly goal)

https://www.mediawiki.org/wiki/User:KSmith_(WMF)/Technology_planned_program_work_Q1#Technology_Program_6._Streamlined_service_delivery

Discussion about work tracking in phab

[edit]
  • Work should be tracked to programs/outcomes/objectives/q-goals
  • Seems most useful to tie to objectives
  • ACTION: Everyone (teams) draft goals for Q2 by next meeting
  • ACTION: Mark will try to figure out a single way for all work within this program to track work
Retrieved from "https://www.mediawiki.org/w/index.php?title=Wikimedia_Release_Engineering_Team/Deployment_pipeline/20170725-planning&oldid=3615671"