Deployment tooling/Cabal/2016-05-23

2016-05-23

Keyholder cleaning

Keyholder cleanup
- Ores somewhat blocked by this: https://gerrit.wikimedia.org/r/#/c/280403/51
- TODO Test in beta :)
- TODO Move public keys from various modules to secrets

Config deploy

Puppet variable file on target
Formatting of config template in repo
Override

Deploying/testing other projects

TODO doublecheck ':' in config values
TODO phab ticket—local server deploying from local machine

Graph Deploy

Example config: https://phabricator.wikimedia.org/P3167
- this is a sample of phabricator's deployment config

List of Doom

Generated with:

find /srv/deployment -type d -maxdepth 2 | sort | nl (and some vim massage)

Services to migrate:

abacist/abacist
Done analytics/aqs
analytics/kraken
analytics/refinery
cassandra/logstash-logback-encoder
cassandra/metrics-collector
Done changeprop/deploy
Done citoid/deploy
Done cxserver/deploy
dropwizard/metrics
dumps/dumps
elasticsearch/plugins
Done eventlogging/analytics
Done eventlogging/eventbus
??? eventlogging/eventlogging
fluoride/fluoride
gdash/gdash (deprecated)
grafana/grafana (deprecated, using debs now on krypton)
graphoid/deploy
iegreview/iegreview
integration/config
integration/kss
integration/mediawiki-tools-codesniffer
integration/php-coveralls
integration/phpcs
integration/phpunit
integration/slave-scripts
jobrunner/jobrunner
kartotherian/deploy
kibana/kibana
librenms/librenms [ops]
logstash/plugins
Done mathoid/deploy
Done mobileapps/deploy
ocg/ocg
parsoid/deploy
Done phabricator/deployment
rcstream/rcstream
restbase/deploy
scholarships/scholarships
sentry/sentry
servermon/servermon [ops]
statsv/statsv
tessera/tessera (deprecated)
tilerator/deploy
wdqs/wdqs
wikimedia/discovery
Done zotero/translation-server
Done zotero/translators

by last fetch date:

tin:/srv/deployment$ ls -latr */*/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  117 Jul 16  2013 fluoride/fluoride/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  119 Feb 27  2014 integration/php-coveralls/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  118 Mar 19  2014 gdash/gdash/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev 4152 May 12  2014 parsoid/Parsoid/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  111 Aug 18  2014 integration/phpcs/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  259 Oct 16  2014 mathoid/mathoid/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  128 Dec  2  2014 restbase/deploy.old/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  121 Dec 11  2014 integration/mediawiki-tools-codesniffer/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  124 Dec 27  2014 abacist/abacist/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  113 Feb 19  2015 integration/phpunit/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  132 Feb 24  2015 dropwizard/metrics/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  120 May  7  2015 kibana/kibana/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  121 May 30  2015 rcstream/rcstream/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  121 Jun 30  2015 tessera/tessera/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  130 Aug 11  2015 logstash/plugins/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  134 Sep 17  2015 elasticsearch/plugins/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  113 Oct  2  2015 integration/slave-scripts/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  252 Oct  2  2015 grafana/grafana/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  113 Oct 22  2015 iegreview/iegreview/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  127 Dec  3 14:52 restbase/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  135 Dec 15 01:59 jobrunner/jobrunner/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  126 Jan  1 19:27 scholarships/scholarships/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  138 Mar 22 16:31 cassandra/logstash-logback-encoder/.git/FETCH_HEAD
-rw-rw-r-- 1 twentyafterfour wikidev  531 Mar 24 00:27 phabricator/deployment/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  226 Mar 31 20:16 eventlogging/eventlogging/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  138 Apr 15 13:32 zotero/translation-server/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  276 Apr 15 13:34 zotero/translators/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  123 Apr 24 20:02 statsv/statsv/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  125 Apr 26 09:08 citoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  360 Apr 26 13:52 dumps/dumps/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  215 Apr 27 07:00 servermon/servermon/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  122 Apr 27 16:06 librenms/librenms/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  252 Apr 28 17:19 kartotherian/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 mforns          wikidev  228 May  2 11:08 eventlogging/analytics/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  226 May  3 13:14 eventlogging/eventbus/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  130 May 10 12:26 mobileapps/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  271 May 10 17:39 ocg/ocg/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  128 May 10 17:58 graphoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  267 May 16 17:34 analytics/refinery/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  129 May 19 18:24 parsoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  115 May 19 21:00 tilerator/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  127 May 20 08:08 mathoid/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  138 May 20 08:15 changeprop/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  128 May 20 11:32 cxserver/deploy/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  141 May 23 11:09 cassandra/metrics-collector/.git/FETCH_HEAD
-rw-rw-r-- 1 trebuchet       wikidev  887 May 23 17:05 wdqs/wdqs/.git/FETCH_HEAD

As Always

Phase 2
Phase 1
Workboard https://phabricator.wikimedia.org/tag/scap3/
Etherpad backed up to https://www.mediawiki.org/wiki/Deployment_tooling/Cabal
Future document https://www.mediawiki.org/wiki/Deployment_tooling/Future
Spreadsheet: https://docs.google.com/spreadsheets/d/1MlEsFxrLvdZdV_G82WEAIvBXr7ArO7nCEKaFClHhJEw/edit#gid=0