Deployment tooling/Notes/New branch deploy

See wikitech:Heterogeneous deployment/Train deploys for live documentation of the Wikimedia deployment process for MediaWiki.

Typical process for deploying a new branch of MediaWiki to the wiki[mp]edia production cluster.

Overview

• Create a new 1.XwmfN branch from current HEAD
• Copy that new version to all MW servers
• Change all Wikipedias to use 1.XwmfN-1 branch
• Change group0 wikis (test/test2/testwikidata/mediawiki) to use 1.XwmfN branch
• Update documentation

Detailed process

1. Run make-wmf-branch script
   1. Branch all extensions
   2. Branch core
   3. Update core branch submodules to reference extension branch HEADs

   This could be done by a Jenkins job or similar automated process

   Should involve a gate process that chooses a stable branch point rather than whatever the state of each HEAD is at the time the process is run

   Should be tested as a unit by regression tests before going to tin

2. Run checkoutMediaWiki on tin
   1. Checkout branch in a new directory on tin
   2. Initialize submodules
   3. Create symlinks to new branch from bits and w to static assets in branch
3. Apply any unmerged security patches from N-1 branch

   Should be automated an incorporated into checkout script

   Should be made to be easier to audit to validate that all patches are present, especially for submodules

4. Git add and commit symlinks
5. Clean up branches on deploy server that have been inactive for >31 days
   1. Cleanup bits symlinks to branch
      • /a/common/multiversion/deleteMediaWiki php-1.23wmfX
   2. Create symlinks cleanup patch
      • git rm -r docroot/bits/static-1.23wmfX
      • git rm -r w/static-1.23wmfX
      • NOLOGMSG=1 git commit -m 'Remove 1.23wmfX symlinks'
   3. Revert change on tin
      • NOLOGMSG=1 git reset HEAD^ --hard
   4. Repeat steps above for each branch as needed
6. Run updateWikiversions
   1. Change wikiversions.json to run all on version N-1
7. Git add wikiversions.json and git commit
8. Run updateWikiversions
   1. Change wikiversions.json to run group0 on version N
9. Git add wikiversions.json and git commit
10. Git push symlink and wikiverisons patches to gerrit
11. Git reset HEAD~4 working directory and index on tin

    Should be scripted

    Should not require ssh agent forwarding on tin

12. Review and approve the symlinks patches in gerrit
13. Wait for zuul & Jenkins to merge patch to branch
14. Git fetch --rebase working copy on tin
15. Delete expired branch checkout in /a/common
    • rm -rf /a/common/php-1.23wmfX
    • Repeat steps above for each branch as needed
16. Manually edit wikiverisons.json to run testwiki on version N
17. Run scap
    1. Validate PHP syntax in config directory
    2. Copy staging directory to common-local on tin (for utility scripts)
    3. Update l10n cache for version N-1
       1. Use PHP to read all l10n files and create CDB for each language
       2. Create json copies of CDB contents
       3. Create MD5 checksums of json copies
    4. Build l10n cache for version N
       1. Use PHP to read all l10n files and create CDB for each language
       2. Create json copies of CDB contents
       3. Create MD5 checksums of json copies
    5. Copy state of tin to sync slaves via rsync pull
    6. Copy state of sync slaves to all MW servers via rsync pull
    7. Build l10n cache on all WM servers
    8. Compile wikiverisons.json to cdb
    9. Copy wikiverions.{json,cdb} to all MW servers
18. Revert local changes to wikiverisons on tin
19. Test testwiki & watch for cluster errors
20. Review and approve the all wikis on version N-1 patch
21. Wait for zuul & Jenkins to merge patch to branch
22. Git fetch --rebase working copy on tin
23. Run sync-wikiverisons
    1. Compile wikiverisons.json to cdb
    2. Copy wikiverions.{json,cdb} to all MW servers
24. Test wikis & watch for cluster errors
25. Review and approve the group0 on version N patch
26. Wait for zuul & Jenkins to merge patch to branch
27. Git fetch --rebase working copy on tin
28. Run sync-wikiverisons
    1. Compile wikiverisons.json to cdb
    2. Copy wikiverions.{json,cdb} to all MW servers
29. Test group0 & watch for cluster errors
30. Create/update deploy notes on mw.o
    1. git checkout --track -b wmf/N origin/wmf/N
    2. git submodule update --init --recursive
    3. php uploadChangelog.php wmf/N
    4. git checkout --track -b wmf/N-1 origin/wmf/N-1
    5. git submodule update --init --recursive
    6. php uploadChangelog.php wmf/N-1
    7. git checkout --track -b wmf/N-2 origin/wmf/N-2
    8. git submodule update --init --recursive
    9. php uploadChangelog.php wmf/N-2

    This should be automated to run every day or after every branch merge

    Should use a bot account for upload

    Upload script should use oauth for authentication

31. Update Roadmap with deploy status