GitLab/Workflows/Security patches

Policy Workflows Project setup assumptions Registering an account on GitLab Hosting a project on GitLab Pipeline conversion Mirroring projects Configuring CI Publishing documentation Code review Making a merge request Squashing Managing security patches Deploying services to production Gerrit to GitLab Localization updates wikitech:GitLab GitLab implementation

The following is a proposed GitLab workflow for managing security/embargoed patches for Wikimedia code. Note that for now, much of the workflow will likely remain the same as the current process, with the addition of using some of GitLab's features to hopefully improve upon capabilities and efficiency.

Reporting, creating and deploying security patches

Initial steps

Do not post security reports or related patches to gerrit, github or any other PUBLIC developer tools, websites, mailing lists, IRC channels, etc. prior to any responsible disclosure of the underlying security issue.

1. Upon discovery of any security issues, please review the information and steps within Wikimedia's Reporting Security Bugs documentation.
2. If you have opted to create a Phabricator security report (as opposed to emailing a report to security@wikimedia.org), you will first need to create a Phabricator account to do so if you do not have one. Then submit a secure report via this form. Submit as much detailed information as possible following the guidelines within the aforementioned Reporting Security Bugs documentation.
3. Develop your patch locally (preferably on a temporary branch) and test it within MediaWiki-Vagrant, MediaWiki-Docker or an an-hoc local development environment. It is recommended to not use a public-facing tool such as patchdemo.
4. Create a patch file within your working directory by running git format-patch HEAD^ --stdout > Txxxxx.patch (where "Txxxxx" is the Phabricator task id).
5. Upload the patch by attaching it to the relevant Phabricator task as a secure file attachment. Coordinate with other developers to review your patch by reaching out to them via email, IRC, etc. and then subscribing them to the security task. Add the Phabricator #Patch-For-Review project to the task.

For code which canonically exists within gerrit.wikimedia.org, please follow the steps below

1. Once again, please do not upload the patch publicly to gerrit for review at this time, unless approved to do so by a member of the Security Team. Backports can happen via gerrit once any relevant security patches have been deployed to Wikimedia production OR when relevant parties have been sufficiently alerted to the issue. For any clarifications on this policy, please contact security-help@wikimedia.org.

For code which canonically exists at gitlab.wikimedia.org, please follow the steps below

Assuming a security patch has been created and reviewed within Phabricator, as per the initial steps mentioned above, please proceed with the following Gitlab-specific steps.

1. If you do not currently have a Wikimedia developer account, you will need to create one and then sign into Gitlab. If you've already created a Phabricator account, chances are you've already completed this step.
2. Once signed into GitLab with your account, click on the Groups > Explore Groups option and then click the New Group button.
3. Create a new group named Security-Txxxxx where Txxxxx is the Phabricator task id of the task you created to track this security issue. Ensure that the visibility of this group is Private. Under the Members section of the group, invite relevant members of the Security Team and others who might assist with code review. Only add as many users as necessary - the fewer users the better.
4. Next, browse the list of projects and their repositories to find those relevant to the security issue.
5. Once you have found and navigated to the appropriate project, click on the Fork icon located in the top-right corner.
6. By default, GitLab will want to fork the repository into your user's namespace. Instead, select the private Security-Txxxxx group listed below the pre-selected user namespace. This may take GitLab some time to complete depending upon the size of the repository. To verify the new fork, examine the url in your web browser - it should contain the path: security-txxxxx/name-of-the-project.
7. Assuming you have a working security patch from the initial steps section, you'll likely want to add any relevant ssh keys to GitLab so you can easily push changes to this newly-forked repository. You can create a new development branch if you'd like, but since this is a forked project under a private group namespace, you can simply work from the main branch.
8. Now apply your security patch within a cloned repository of your private forked repository. You may also add relevant individuals, including members of the Security Team, to further review or monitor the security patch within Gitlab. Let any relevant CI/CD pipelines run to validate the code prior to any additional approvals or merges back to the main project.
9. Once the security patch has been tested and deployed to production (see manual steps below), a merge request should be created to merge all relevant changes from the security issue project fork back to the original project's repository. This should involve a merge from a main branch to a main branch. This can be accomplished by choosing the Merge Requests menu item within the right navigation and completing the steps. You should compare the changes and select the option to squash commits when the request is accepted. For the merge request title and/or description, SECURITY: fix for Txxxxx should suffice. This process can also be used to backport to any relevant release branches in addition to the main branch.
10. An important reminder: do not create or attempt to merge a merge request until the security issue has been properly disclosed, which a member of the Security Team should be able to verify.
11. Once the merge request has been approved and merged, the forked project and private group should be deleted. For the project, this can be accomplished by navigating to Settings > Advanced > Expand > Delete project. For the private group, this can be accomplished by navigating to Groups > Your Groups > the relevant private group > Settings > Advanced > Remove group.

Other systems

1. For code which canonically exists within github, please follow the steps within Wikimedia's Reporting Security Bugs documentation and do not create any public pull requests.
2. If the relevant code repositories exist canonically within another versioning system or repository, please contact security-help@wikimedia.org for additional guidance.

Deploying security patches to production

1. Once the security issue has been privately discussed within Phabricator and all relevant security patches have been tested, reviewed and approved, they should be uploaded to the current deployment server.
   • Check that the patch applies with git apply --check /path/to/patchfile on the relevant deployment branch(es).
   • Apply the patch with git am /path/to/patchfile to the relevant deployment branch(es).

   Note that some config files are made public via noc.wikimedia.org; don't put anything non-public within those.

2. Deploy as usual but use --no-log-message to prevent the automatic logging from revealing too much information regarding affected files or components. Log the deployment manually by saying "!log Deployed patch for Txxxxx" in #wikimedia-operations ^connect.
3. Ensure the security patch will be applied to future deployment branches:
   • The .patch file should be stored on the deployment host under /srv/patches/<branch>/, for whichever branches have the security patch applied.
     • Patches to MediaWiki core should be stored in /srv/patches/<branch>/core/
     • Patches to extensions should be stored under /srv/patches/<branch>/extensions/ExtensionName.
       • Filenames should be prefixed with a 2-digit number to indicate the order in which patches should be applied in the repo. Your file should be prefixed with '01-' if it is the first patch in the directory, or the next highest number if other patches already exist.
       • Patch files should be git-committed to the local repository within /srv/patches/.
   • Add a note or link to a log entry to the relevant Phabricator task noting that the security patch has been deployed.
   • If the Security Team isn't already aware of this particular security issue, be sure to inform them that you deployed the patch to prevent duplicate effort.
4. Work with the Security Team to make sure the vulnerability is resolved and that your patch makes it into the next security release.
   1. If there is no PII or other sensitive data within the associated Phabricator task, it can now be made public. If you are unsure how to do this or if the task should be made public, please consult a member of the Security Team.
   2. Perform any necessary backports within the relevant versioning systems (gerrit, GitLab, etc.) to supported release branches assuming the patch is relevant for and applies to previous versions. As an extra step, once the branch is backported to master, it can be removed from /srv/patches/ as it will no longer apply to future production release branches. This will make Release Engineering's life a little easier.
   3. Request a CVE, if appropriate. Note the CVE ID within the task's title, description and within any related security release tasks.

Potential deployment problem 1: Submodule security patches not committed

Sometimes you may find a security patch for an extension that has not been committed:

[you@deploy1001 php-1.999.0-wmf.1 (wmf/1.999.0-wmf.1 * u)]$ git status
On branch wmf/1.999.0-wmf.1
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   extensions/InsecurityExtension (new commits)

Submodules changed but not updated:

* extensions/InsecurityExtension fffffff...eeeeeee (1):
  > SECURITY: Make the Insecurity extension secure

no changes added to commit (use "git add" and/or "git commit -a")

This is normal! The reason for this state is that subsequent updates to the submodules require human intervention to fix merge/rebase conflicts.