Codesearch/Admin

codesearch currently runs from the "codesearch" Cloud VPS project.

openstack-browser

Grafana: Cloud VPS, to monitor the health of the instance.

Architecture

Each search profile (everything, extensions, skins, etc.) listed are backed by individual instances of Hound, that are controlled by hound-$backend systemd units. Hound is a go application that is run in a Docker container using an upstream-provided image (TODO: build our own image). A flask web application proxies requests to the individual Hound backends and injects in our customizations. It runs through the hound_proxy systemd unit.

Everything is puppetized, so puppet will ensure all of the systemd units are running. Note that all of the backend units depend upon the proxy, so restarting the proxy will restart all of the backends.

The larger backends can take up to 15 minutes to start. The start up process can be rather memory and CPU intensive (especially on a truly cold start on a brand new instance), so the wait.py script tries to only allow one backend to start up at once.

The health check page should be helpful in seeing whether the Hound backends are actually down, or just starting up still.

The configuration for each backend is written by the codesearch-write-config systemd timer. Each backend's unit is configured to restart every 24 hours, so it picks up the new configuration.

Adding a new search profile/backend

In labs/codesearch:

Update write_config.py to generate configuration for the new backend
Update app.py with a label for the search profile
Example: gerrit:565424

In operations/puppet:

Update hieradata/cloud/eqiad1/codesearch/common.yaml with the new backend and the next numerical port
Example: gerrit:567190

The codesearch patch must be merged before the puppet one. Also ensure that the codesearch-write-config systemd timer has run before the puppet patch is merged.

Upgrading hound

Codesearch is a wrapper around hound. We deploy Hound using a container image built by the deployment pipeline. To upgrade Hound:

In a checkout of the labs/codesearch Git repo:

$ git submodule update --init
$ cd hound
$ git checkout v0.X.X
$ cd ..
$ git commit -a -m "Upgrading Hound to v0.X.X"

Merge the patch in Gerrit, which will trigger the deployment pipeline to build a new image. Once that's done, log into the codesearch instance and run:

$ sudo docker pull docker-registry.wikimedia.org/wikimedia/labs-codesearch:hound
$ sudo systemctl restart hound_proxy # Trigger a rolling restart of all instances

Deploy frontend

Log into the codesearch instance and run:

$ sudo -i
root$ cd /srv/codesearch/
root@codesearch$ sudo -u codesearch git pull --rebase
root@codesearch$ git log # review
root@codesearch$ cd frontend/

root@frontend/$ docker build . -t codesearch-frontend && systemctl restart codesearch-frontend