Wikimedia Product/Wikimedia Product Infrastructure team/Service deployment process
Product Infrastructure maintains a number of Node.js services running on Kubernetes via the Wikimedia deployment pipeline. This page describes the process for deploying new versions of the services.
In a nutshell: when a patch in Gerrit is merged to the master branch of a service repo, a post-merge CI job is run to build a Docker image, according to the definition in the project's Blubberfile, from the updated master branch. Deploying the service simply means updating the service configurations on deployment.eqiad.wmnet and any Beta Cluster instances with the resulting image tag.
Bumping apt package versions
[edit]In order to achieve that, you need to:
- Send an empty commit to trigger a new image build with the latest apt packages
- Update deployment-charts accordingly
- Follow the production deployment steps in the following section
Production
[edit]Helm charts defining the production service configurations are stored in the operations/deployment-charts repo. To deploy a new version to production, a patch must be created and merged updating the per-cluster service definitions under helmfile.d/ with the Docker image tag created in the post-merge service-pipeline-test-and-publish job. Three deployment clusters are defined: staging, eqiad, and codfw (with the latter two corresponding to Wikimedia's two core server clusters). An example patch updating each with a new Docker image tag for deployment is here. These patches can be self-merged by the deployer at deploy time. After merging, allow a minute or two for the repo on the production server to be synced with the master branch.
The next step is to sync the change to the Kubernetes cluster. On our production deployment servers (accessible as deployment.eqiad.wmnet) the deployment-charts repo lives at /srv/deployment-charts. Navigate to the service's helmfile directory for the staging cluster, examine a diff of the changes, and if the diff contains no surprises, sync the change:
$ ssh deployment.eqiad.wmnet
$ cd /srv/deployment-charts/helmfile.d/services/<service>/
$ helmfile -e <environment> diff
$ helmfile -e <environemnt> apply
$ # check the pods status
$ kube_env <service> <environemnt> && kubectl get pods
If everything looks OK, repeat the process in /srv/deployment-charts/helmfile.d/services/eqiad/<service>/ and /srv/deployment-charts/helmfile.d/services/codfw/<service>/.
NOTE: <environment> is one of the following as declared in the helmfile.yaml for each service: staging, eqiad, or codefw.
Beta Cluster
[edit]Deploying to the Beta Cluster involves a similar procedure. Log into Horizon, navigate to the deployment-prep project, select Compute > Instances from the left-hand nav menu, and click on the name of the instance to which you intend to deploy. In the instance view, click the Puppet Configuration tab and observe the Hiera Config section of the resulting screen. In the configuration shown, you should see a profile::docker::runner::service_defs setting containing a familiar-looking service-runner service configuration; find the version key under profile::docker::runner::service_defs and update it with your new image tag. The instance will pick up the updated configuration during the next Puppet run.
See also
[edit]- Wikimedia Apps/Team/RESTBase services for apps/Deployment process — historical deployment process