Git/Conversion/status

Last update on: 2013-07-monthly

2011-09-22

Rob Lanphier proposed a draft plan on the wikitech-l mailing list.

2011-09-30

Rob Lanphier proposed a draft plan to proceed to a migration to git over the following months, and got a positive response from developers.

2011-10-31

Chad Horohoe started to prepare for the migration to git; he wrote a script to compare LDAP accounts to the information stored in Code review, and prepared for conversion tests from the current Subversion repository. An important step in the conversion is having names and email addresses for all committers, since neither Git nor Gerrit function well without them. As of this writing, 26 committers do not have any contact information, and 25 have a name but no e-mail address. Please help by filling in the names of committers you know.

2011-11-16

Chad is down to one last email address of a committer to confirm, then a full import will happen. He has already started a partial import (up to r100000), and will do the complete import once that is done.

2011-11-30

Chad Horohoe got a lot of help from the community on identifying unknown committers; user mapping is now complete. Gitorious was installed on a virtual machine, and a test git conversion of the MediaWiki code repository is now imminent. Meanwhile, there have also been discussions about e-mail aliases for LDAP users, to avoid disclosing private e-mail addresses. Chad and Brion Vibber worked on changes to the development workflow introduced by the move to git (e.g., when continuous integration tests get run).

2011-12-19

Have currently working conversion of MediaWiki core in Gerrit and are sorting out the permission model. We're having two meetings this week (both on Dec. 19th) to help hammer out the details. Join us in #wikimedia-dev!

2011-12-31

The rules to convert phase3/ have been finished and a test repository is now available. Chad Horohoe is now working on convert rules for extensions, and on user permissions.

2012-01-31

The MediaWiki phase3 repository has been successfully converted with branches. Another upcoming test repository will include release tags, and extension projects will be set up shortly.

2012-02-23

The migration of MediaWiki core and of extensions used on WMF sites is now scheduled for the first weekend in March (3 March 2012). Sumana Harihareswara and Chad Horohoe wrote a blog entry to answer some common questions about the migration. Starting on 24 February WMF Engineering will start making strong efforts to train the development community in using Git and Gerrit, including screensharing trainings.

2012-02-29

The migration of MediaWiki core and of extensions used on WMF sites is now tentatively scheduled for 21 March 2012. Sumana Harihareswara and Chad Horohoe wrote a blog entry to answer some common questions about the migration. WMF Engineering is making strong efforts to train the development community in using Git and Gerrit, including documentation and trainings via screensharing on February 27 and 28. We will be moving Gerrit infrastructure to Ashburn, as the Tampa server we're using can't handle the load.

2012-03-22

We've now moved MediaWiki core and WMF-deployed extensions over to Git and Gerrit, and for those directories Subversion is now read-only. We've communicated links and workflow planning, and the new procedure to add and remove people from Gerrit project owner groups.

2012-03-31

We've now moved MediaWiki core and WMF-deployed extensions over to Git and Gerrit, and for those directories Subversion is now read-only. We've communicated links and workflow planning, and the new procedure to add and remove people from Gerrit project owner groups. A summary of the move was published in the Wikipedia Signpost.

2012-04-13

MediaWiki core and all WMF-deployed extensions are getting into regular development cycles. The first batch of non-WMF extensions was migrated on 6 April, with only minor issues. Many small issues in Gerrit/Git have been fixed over the past week. Notably, automated localization commits have been fixed and will resume on Monday, April 16. On April 10, we cut the first WMF deployment branch (1.20wmf1) and pushed it to test2.wikipedia.org and mediawiki.org, with success. The priorities for Gerrit are currently fixing our UTF-8 problem, improving Gerrit integration with IRC/e-mails, making project/user information more discoverable, and upgrading Gerrit to 2.3 in the next 2 or 3 weeks.

2012-04-monthly

MediaWiki core and all extensions deployed on Wikimedia sites are getting into regular development cycles. The second group of extensions were migrated on April 27. We've also branched and released 1.20wmf1 to most Wikimedia sites. The priorities for Gerrit are currently fixing our UTF-8 problem, improving Gerrit integration with IRC and e-mail, making project/user information more discoverable, and upgrading Gerrit to 2.3 in the next week or so.

2012-05-06

We successfully upgraded to Gerrit 2.3 on Friday, May 4th. This brings a variety of fixes and features--notably, a less cluttered diff interface and the ability to have the "mediawiki/extensions" meta-repo that holds all extensions be automatically updated. Several new repositories were created for extensions and other uses, including a repository for packaging MediaWiki to easily install on Windows servers. We also managed to fix our long-standing UTF-8 issue so users can all now use unicode for their commits, comments and usernames without fear (there is one minor issue remaining with IRC output). The coming week will focus on improving IRC/logging output, and fixing the auto-commits of localization updates from Translatewiki.

2012-05-monthly

Chad Horohoe and Ryan Lane upgraded Gerrit to version 2.3, which brings a variety of fixes and features; notably, a less cluttered diff interface, and the ability to have the "mediawiki/extensions" meta-repository that holds all extensions be automatically updated. Many new repositories were created for extensions and other uses, including a repository for packaging MediaWiki to easily install on Windows servers. We have also managed to fix our long-standing UTF-8 issues with help from Marcin Cieślak, so users can all now use Unicode for their commits, comments and usernames. Image changes can now be shown via the UI, rather than being downloaded as a ZIP file to be compared locally (example). Brion Vibber has agreed to lead a process for evaluating Gerrit (and possible alternatives), which will conclude in early August. David Schoonover is currently writing up a list of alternatives to Gerrit which he plans to publish on mediawiki.org and announce on wikitech-l.

2012-06-21

Created a bunch of new repositories in Gerrit this week--including all extensions that are enabled on Translatewiki.net. Gerrit 2.4.1 has been released, and have begun testing it so we can upgrade our installation.

2012-06-monthly

This has been a very busy month for Gerrit. The creation of new projects continues; this month saw every extension deployed on Translatewiki moved to Git. During the week of June 25th, we experienced some downtime with Gerrit due to search engine crawlers overloading the server. Also that week, lots of improvements to IRC logging were made, although discussion continues on how to make the bots more effective. We have scheduled an upgrade of Gerrit to the 2.4 release for the week of July 2nd--this will bring the much desired "Rebase button" to Gerrit which should lighten users' workload for trivial merges.

2012-07-monthly

We're in the process of evaluating Gerrit and its alternatives as a code review tool. Chad Horohoe and Ryan Lane upgraded our Gerrit instance to 2.4, which provided many incremental fixes and small features (e.g. the "rebase" button). Ryan and Asher Feldman migrated the Gerrit database to our Ashburn datacenter, which resulted in a big performance boost.

2012-08-monthly

Chad Horohoe spent a good amount of time fixing issues upstream, including two big improvements to the project listing page. He also cleaned up the Gerrit installation on Labs to more accurately mirror production—also cleaning up the production setup along the way. Initial research was done into replication to GitHub. Finally, Gerrit 2.5 is nearing release, which brings a bunch of new features (like plugins) and fixes. The Labs instance of Gerrit is already running the release candidate. In September, we'll be upgrading to Gerrit 2.5 and getting repositories replicated out to GitHub.

2012-09-monthly

September saw some major problems with Gerrit that took about two full weeks to repair. Replication testing and preparation has completed, and we will begin replication to Github in early October. Gerrit 2.5 is nearing release and has been tested, which we will be upgrading to in early October. Finally, discussion has begun about what to do with the code still in Subversion.

2012-10-monthly

Many more extensions are now replicated from Gerrit to the Wikimedia account on GitHub. Gerrit 2.5rc1 and rc2 were released over the course of October, and a final 2.5 is expected soon. One particularly exciting feature in Gerrit 2.5 is the new extensibility framework, which will allow us to replace our gitweb based source browser with GitBlit (the latter being a far more usable code browsing option). Assuming a showstopper bug with LDAP propagation gets fixed, we'll be able to deploy Gerrit 2.5 shortly after its release, and GitBlit shortly after that.

2012-11-monthly

We're still very much looking forward to deploying the latest version of Gerrit (see last month's update), but unfortunately remain blocked on a complicated LDAP propagation issue. Chad Horohoe is working with the Gerrit developers on finalizing the fix for this issue. Chad also attended the Gerrit Developer Summit in November, and both Chad and Rob Lanphier attended the Gerrit Users Summit (notes).

2012-12-monthly

There's not much to report for the month of December so far with Gerrit. New repositories continue to be created, and the vast majority of active parts of SVN have been marked read-only by now. Upgrading to a newer version of Gerrit is still blocked on our LDAP problem with master, but the patch to fix that is nearly complete. Mid-December, we extended the Verified category to now allow +2 (in addition to +1 and -1), so Jenkins has a wider range of statuses it can report.

2013-01-monthly

The ExtensionDistributor was rewritten in early January. While this was primarily done to support the data center migration, this was the first time ExtensionDistributor had received any signification attention since the migration to Git. The new version now utilizes the Github API to generate extension snapshots. We hope that the new version will be more reliable for users. SVN-based extensions are no longer supported, but this is not expected to impact many users since these extensions are largely unmaintained (all popular and active extensions have long since moved to Gerrit). As always, these extensions will remain in SVN should anyone still want the code.

2013-01-24

The MediaWiki SVN repository was made read-only on January 25, 2013. Gerrit 2.6 upgrade is coming soon, with the final upgrade issues being ironed out. Also, work has begun on tighter integration with Bugzilla for Gerrit, slated for soon after we upgrade.

2013-02-monthly

Gerrit was upgraded this month to a pre-2.6 snapshot. This enabled the use of plugins, as well as brought numerous bugfixes and UI improvements. Work is underway on a plugin to provide Bugzilla integration and to replace Gitweb with a better repository viewer called Gitblit. All of our git repositories are now automatically replicated to GitHub. We've begun some initial planning into how we can improve the "new repository request" process, making it much easier for users with a quicker turnaround time.

2013-03-monthly

This month saw a minor update to our Gerrit installation, bringing more bugfixes and features to our developers, as well as a finalized version of the project deletion plugin. A release is planned for the very end of the month to deal with some long standing performance issues on larger repositories (like MediaWiki core). The Bugzilla integration plugin is undergoing a second iteration, based on feedback from the community during the first deployment. Work is also underway for project renaming, as well as making project creation easier for users.

2013-04-monthly

We deployed a first iteration of a Bugzilla integration plugin, which provides notifications to Bugzilla when changes are made in Gerrit. We’ve increased the memory allocated to Gerrit, as well as deployed a couple of other stability fixes; both of these changes should provide some minor performance and stability improvements to users. Finally, we’ve deployed a new version of Gerrit that includes superior garbage collection support. This drastically improved the compression of repositories on-disk, which has resulted in a wide range of improvements for all users for all operations, from cloning to pushing to commenting on changes.

2013-06-monthly

Chad Horohoe and Christian Aistleitner upgraded our Gerrit instance from a pre-release version of 2.6 to a pre-release version of 2.7 on the last week of June. They've additionally published a new version of the Bugzilla/Gerrit integration plugin. Details about new functionality can be found in the Gerrit 2.7 draft release notes.

2013-07-monthly

With the migration of pywikipediabot from Subversion to Git, we were able to switch svn.wikimedia.org to read-only mode, thus completing this migration. We plan to keep the Subversion service around indefinitely for archival purposes, and can still migrate any dormant project that hasn't already been migrated on request.