Deployment

Getting your service ready to be deployed on WMF production machines involves several tasks. This document explains the steps needed to get started and how to keep your deployable copy up-to-date.

Repositories

Because Node.js services use npm dependencies which can be binary, these need to be pre-built. Therefore, two repositories are needed; one for the source code of your service, and the other, so-called deploy repository. Both should be available as WM's Gerrit repositories with the paths mediawiki/services/your-service-name and mediawiki/services/your-service-name/deploy. When requesting them ask for the former to be a clone of the service template and the latter to be empty.

It is important to note that the deploy repository is only to be updated directly before (re-)deploying the service, and not on each patch merge entering the master branch of the regular repository. In other words, the deploy repository mirrors the code deployed in production at all times.

The remainder of the document assumes these two repositories have been created and that you have cloned them using your Gerrit account, i.e. not anonymously, with the following outline:

~/code/
  |- your-service
  -- deploy

Furthermore, it is assumed that you have initialised the deploy repository:

$ cd ~/code/deploy $ git review -s
$ touch README.md $ git add README.md
$ git commit -m "Initial commit"
$ git push -u origin master  # or git review -R if this fails
# go to Gerrit and +2 your change, if needed and then:
$ git pull

Finally, if you haven't yet done so, do basic service configuration.

The remainder of the document refers to these two repositories as the source repository and the deploy repository, respectively. Note, however, that your actual local paths may differ. If that is the case, simply use your local paths according to the type of repository the document is referring to.

Configuration

The service template includes an automation script which updates the deploy repository, but it needs to be configured properly in order to work.

package.json

The first part of the configuration involves keeping your source repository's package.json updated. Look for its deploy stanza. Depending on the exact machine on which your service will be deployed, you may need to set target to either ubuntu or debian.

If you want to specify a version of Node.JS, different from the official distribution package, set the value of the node stanza to the desired version, following nvm versions naming. To explicitly force official distribution package, "system" version can be used.

The important thing is keeping the dependencies field up to date at all times. There you should list all of the extra packages that are needed in order to build the npm module dependencies. The _all field denotes packages which should be installed regardless of the target distribution, but you can add other, distribution-specific package lists, e.g.:

"deploy": {
  "target": "ubuntu",
  "node": "system",
  "dependencies": {
    "ubuntu": ["pkg1", "pkg2"],
    "debian": ["pkgA", "pkgB"],
    "_all": ["pkgOne", "pkgTwo"]
  }
}

In this example, with the current configuration, packages pkg1, pkg2, pkgOne and pkgTwo are going to be installed before building the dependencies. If, instead, the target is changed to debian, then pkgA, pkgB, pkgOne and pkgTwo are selected.

Three options are available for installing packages:

Simple install. Include the package name as a string like in the example above and the package will be pulled from default debian repositories.

Install from custom source. Specify an object describing the installation source and list of packages to install from that source. Example:

{
    "repo_url": "https://apt.wikimedia.org/wikimedia",
    "release": "jessie-wikimedia",
    "pool": "main",
    "packages": [
        "librdkafka-dev"
    ]
}

Install deb from URI. To pull the package from a custom location and install it via dpkg use the following syntax:
```
{
    "url": "https://apt.wikimedia.org/wikimedia/some_package.deb"
}
```

As a rule of thumb, whenever you need to install extra packages into your development environment for satisfying node module dependencies, add them to deploy.dependencies to ensure the successful build and update of the deploy repository.

You can provide the "install_opts" key and specify additional options to the npm install command.

Local git

The script needs to know where to find your local copy of the deploy repository. To that end, when in your source repository, run:

git config deploy.dir /absolute/path/to/deploy/repo

Using the aforementioned local outline, you would type:

git config deploy.dir /home/YOU/code/deploy

The source repository is itself a submodule of the deploy repository. If its name as specified in package.json's name field does not match the actual repository's name in Gerrit, run:

git config deploy.name name_in_gerrit

That will make the system look for the repository mediawiki/services/name_in_gerrit when checking it out in the deploy repository. If, however, you do not use MediaWiki's Gerrit installation to host your repository, you can specify a different one using the deploy.submodule configuration directive by supplying the full remote reference URL:

git config deploy.submodule https://github.com/your_team/repo_name

The deploy-repo builder script assumes the name of the remote to check out in the deploy repository is origin. An alternative name can be configured by invoking (in the source repository):

git config deploy.remote deploy_repo_remote_name

You may also need to build the deploy repository using a source repository branch other than master. To do that, tell git the name of the branch with:

git config deploy.srcbranch src_repo_branch_name

Likewise, if you are using the same deploy repository for different production clusters, you can instruct the build script to use a deploy repository branch other than master with:

git config deploy.deploybranch deploy_repo_branch_name

Note that if you change the source and/or deploy repository branch name, the build script expects them to exist in Gerrit and they have been properly set up.

Testing

Before updating the deploy repository you need to make sure your configuration works as expected. To do that, in your source repository run:

./server.js docker-test

The script will build a new Docker image, install the needed packages and npm dependencies and run the test suite. Tweak your code and configuration until everything works as expected (and commit those changes).

Update

The final step is updating the deploy repository. First, make sure that your source repository has got the latest dependencies installed:

rm -rf node_modules/ && npm install service-runner

Update the deploy repository by running from the source repository:

./server.js build --deploy-repo

The script will:

create the proper deploy repository outline
fetch the updates
ensure the submodule is present
update the submodule
build the npm dependencies
commit the changes with a pretty-formatted message

There is also a handy shortcut for sending the patch to Gerrit immediately. To do so, add the --review argument to the call:

./server.js build --deploy-repo --review

Note that if no changes were made to the source repository, the script aborts its execution. If, nevertheless, you need to rebuild the dependencies, you can do so using:

./server.js build --deploy-repo --force