GitLab/Initialization/Decision Log
< GitLab | Initialization
| Date | Category | Decision | Why? | Comments |
| 2/23/2021 | Management | Keep tasks and most of communication in Phabricator, the board https://phabricator.wikimedia.org/project/view/5212/ | Easier tracking | |
| 2/23/2021 | Management | Create documentation of the process | To know why the particular decision was made and what files were changed, to be able to reproduce the steps | |
| 3/1/2021 | Security | SSO project provides the CAS service to other services in development. As such please only use your access for creating/updating ldap accounts. Do not change any of the CAS configuration. If there are some CAS changes you want to test please let us know here and we should be able to turn things around quite quickly. https://phabricator.wikimedia.org/T274461 | ||
| 3/3/2021 | Tools | Backups: NFS works, but machines need access to where that goes.
Bacula - local backup would be good then we could point the bacula daemon Gitlab backup has to be consistent. Put on local disc, then copied and stored for later. Will grow over time. Backups task: https://phabricator.wikimedia.org/T274463 |
||
| 3/4/2021 | Tech solutions | Prometheus metrics, Grafana dashboards, and logging to Logstash have to be a part of the current scope of work | https://prnt.sc/10p3h69 | |
| 3/4/2021 | Tools | In our current case can we consider the VM in WMCS/horizon (or potentially a new VM in WMCS/horizon) as the dev environment. Then we need a second machine in our prod environment only when a load induced split is necessary. | WMCS is a real testing environment with a minimal amount of scrutiny. It is the correct environment for developing a test. At the end it makes sense to run the developed test in the production environment to see what the real behavior looks like, but development is better done in WMCS/horizon | |
| 3/4/2021 | Tech solutions | Starting with performance testing, the results: https://docs.google.com/spreadsheets/d/1OJmkPKsQvb3YmIaG-lw7oyzZnnGxEqkCxJSPnHD1l-0/edit#gid=389161915 | ||
| 3/8/2021 | Tools | Using GPT (GitLab Performance Tool) | GPT works pretty well. We use it as a base for the test scenarios. GPT is available to run on a VM | |
| 3/9/2021 | Tools | Using Let's Encrypt for maintaining certificates | WMF uses Let's Encrypt with its servies | |
| 3/10/2021 | Tools | Using k6 performance testing tool | Minimizing time and effort to start (and subsequent maintenance), it may make more financial sense to pay for it rather than to expand the project scope. The difference will be fading with time, so maybe we can explore a mixed approach: start with a service now, then migrated to a self-hosted k6 instance.
Support for (many) geographically distributed load agents, this results in a more realistic numbers, but I'm not sure how much value this brings to you guys. Support for test data retention, visualization and analysis, what we briefly discussed before. |
|
| 3/10/2021 | Tech solutions | Gitlab DNS: gitlab.wikimedia.org service alias, point to gitlab1001 | https://phabricator.wikimedia.org/T276170 | |
| 3/16/2021 | Tools | Using Ansible for installing GitLab (not Puppet, and not manually) | Ansible will help us to automate the routine process of Gitlab installation and system configuration, not clashing with Puppet that WMF uses to maintain hosts configurations | |
| 3/16/2021 | Tech solutions | For performance testing - let's skip CAS | Not to have multiple user creation | |
| 3/22/2021 | Tech solutions | Create a new test instance for Ansible tests | Not to break current server | |
| 3/30/2021 | Tech solutions | Performance acceptance metrics: api_v4_users, web_project, git_push, git_pull; Whole test suite as an generalized view of the Gitlab performance | These metrics are representative of important use cases and general server performance | |
| 3/30/2021 | Tech solutions | MediaWiki/core would be an excellent test case ssh://<username>@gerrit.wikimedia.org:29418/mediawiki/core.git | It's one of our larger repos and it is one of the more accessed-by-humans repos | |
| 4/5/2021 | Strategy | Gitlab configuration through Web UI (or equivalent API) is not included in work by S&F, will be handled by Release Engineering | Anything that is not configured in gitlab.rb is going to be maintained by the SRE team manually at this stage; In the future configuration management through API is possible | |
| 4/5/2021 | Tools | gitlab.rb will be version controlled in ansible repo on S&F gitlab | For the flexibility on reconfiguring on different deployments | |
| 4/5/2021 | Tech solutions | Testing over HTTPS is good enough (donât have to do SSH) | git over HTTPS performance is representative of expected git over SSH performance, while developing native SSH tests seems resource expensive at the moment | |
| 4/5/2021 | Tech solutions | If you decide to dynamically resize the gitlab instance, without restarting - donât forget to run sudo gitlab-ctl reconfigure | It will self-configure itself for new parameters | |
| 4/5/2021 | Tech solutions | Separate admin SSH and Gitlab SSH daemons on different IP addresses | For added security: Gitlab SSH will only allow gitlab user, will be configured for higher security and performance | |
| 3/10 | Tech solutions | Production instance certificates are managed by acme-chef, separately from Gitlab | Allows for better WMF control over certificates and monitoring (https://phabricator.wikimedia.org/T276673) | |
| 5/10/2021 | Tech solutions | SSO/CAS decided to be the authentication method for Gitlab | CAS is a hard requirement from the SRE team and it doesn't create unacceptable functional limitations at the moment | |
| 5/4/2021 | Tech solutions | Backups: Daily gitlab-backup backups for MVP | More frequent incremental or snapshot backups research postponed after the MVP roll out | |
| 3/23/2021 | Tech solutions | Mail configuration outbound email is set to use a local puppet-managed MTA, inbound email is not supported, but planned for the future | ||
| 5/7/2021 | Tech solutions | Logging: puppet-managed Logstash ingests a list of SF-suggested logs into ElastiSearch | ||
| 4/10/2021 | Tech solutions | GiLabâs specific SSHD should have separate host keys and separate env configuration file | GilLab's specific SSHD settings should'nt incluence management SSHD | |
| 5/17/2021 | Tech solutions | SSO/CAS decided to be the only way of authenticating users to Gitlab | Per a call with the WMF team 5/17, extra authentication methods provide little value at the moment | |
| 5/17/2021 | Tech solutions | SSO/CAS users are to be "internal" type of users in Gitlab terms | WMF strives to treat contributors the same way as employees, not much value in this distinction | |
| 5/3/2021 | Tech solutions | Attributes to import from SSO on authentication: 'CN' (for username and name), 'mail' for e-mail address | ||
| 6/21/2021 | Tech solutions | Bacula Backups: use Full backups instead of Incremental | GitLab is doing full backups in a self-contained .tar arcive every day. To reduce the complexity and disk usage for Bacula and represent the actual backup strategy we decided to use daily full backups. We decided against Incremental backups, because this would introduce a technical dependency between daily GitLab backups although there is no real dependency. At a later stage we may use incremental backups to snapshot git repositories or databases, but for MVP daily full backups are the best option. See https://wikitech.wikimedia.org/wiki/GitLab for more detail. |