Here are some features of Gerrit our team use regularly that we would very much like to see in any future system:
> Being able to easily create a stack of dependent commits, each one representing a single commit in the final tree, and being able to modify those commits, and review changes between those modifications
Yes, please! I miss that feature so much, especially when CI flatlines under the load of the 10 patches I submit in a chain. :-)
+1. Our current CI handles quite complicated dependencies, ie: introduce new API in core, patch dozens of extensions to use this new API (depends-on: first patch), then deprecate and remove the original API (depends-on: all the deployed extension patches).
My own working style on "feature branches" also involves incrementally factoring out and rebasing necessary-but-independent fixes to other places in the code as earlier patches in the stack. Github's review process has always made viewing the history of and reviewing such patch stacks very awkward.
Depends-On exists (in some form) in GitLab, unfortunately it's in the enterprise edition (EE). On the Features page we talked about re-implementation via CI.
Mostly a +1 to ESanders comment, I extensively use the ability to create a stack of commits that are separately reviewed, and re-ordering that stack is not so uncommon. My experience attempting this sort of workflow in GitHub (no GitLab experience, but it seems pretty similar) was pretty bad, but that could be user ierror.
+1 to all of the above
SRE tend to break down important changes in subsequent patchsets to better control application of a change (so for instance - first on one server, then on the canary pool, then everywhere).
If we lose the ability to do so with ease, and to rebase/resubmit those stacks of changes, our workflow will need to change radically compared to what we have today.
We also need the ability to apply such stacks of changes to a working tree with ease (for instance for the puppet compiler).
>On the Features page we talked about re-implementation via CI.
This somewhat irks me.
I understand not wanting to necessarily pay GitLab for these features, but developing them in the first place and then debugging and maintaining them is not free either. I doubt they're going to take them back upstream, so we're left maintaining them long term. We might find some other people to contribute, but maybe we won't.
It seems odd to switch tools to make things some things easier, then because we lose some fairly major features, we then reinvent the wheel. Aren't these custom changes and the "wikimedia only" ways of doing such things part of the current issues with CI as is?
I was wondering whether we could use the "merge request dependencies" thing in the premium version to fake up the stack-of-commits model by having the a bunch of merge requests on a single project, which all depend on each other. However, it explicitly doesn't support "a “deep”, or “nested” graph of dependencies", so even if we had it, it might not actually work for that.
That said, assuming that merge requests function approximately the same as github merge requests, we can get much of what Ed wants... if we agree that feature-branches can have changes force-pushed to them. That gets us the stack and the reordering, but not the individual review/merge of commits.
I also agree that developing our own set of complicated behaviors on top of our new tool does seem like we're just jumping head-first back into the issues we're apparently leaving gerrit to avoid.
We might find some other people to contribute, but maybe we won't.
I think this is a good place to mention (or reiterate) that there's a set of projects with values and goals broadly aligned to ours using GitLab's Community Edition. We've reached out to organizations including Debian, KDE, Gnome, and freedesktop.org about their implementations, and generally received a very encouraging response. There's shared technical work being done in this space and (hopefully) an opportunity to collaborate with a broader community.
I think it's worth mentioning here that Github PRs technically do support multi-commit branches, but that's used to reflect changes made in response to review, and the commits are squashed into one before being merged. This means any solution that addresses the "stack of multiple commits" use case also needs to address the "amendments in response to code review" use case. You also need to be able to rebase a change onto an update version of the change it depends on.
I suppose it's possible that some of these things are already supported, and support can be added for the others, but I'm somewhat skeptical that this will work as well as it does in Gerrit.
I think it's worth mentioning here that Github PRs technically do support multi-commit branches
To elaborate on this a bit, multi-commit branches are the long-time default on GitHub (or I should say in the PR/MR model), and while it's my impression there's been more of a trend towards squash-and-merge, it's not abnormal in many projects to develop and land feature branches with multiple commits - be that in order of work done & feedback received, or rewritten into logical units of work. Preferences about the structure of commit history vary considerably in the wild. Generally those preferences can be modeled within a workflow where branches are first-order units of work (GitHub, GitLab), just as they can in one where iteratively re-written patches are the core abstraction (Gerrit).
GitLab's UI lets the submitter or merger check a box to squash commits and edit the commit message. I think that a single commit is the appropriate result of most merges from a branch, but there are probably exceptions, including some of the simple cases for a chain of dependent commits. (As a builtin abstraction for grouping changes, branches do have real affordances here that, if not precisely lacking in Gerrit, are at least located differently.)
I note that GitLab merge review comments appear to persist through a force-pushed rewrite of history on a branch, so that's an option for responding to feedback where rewritten commit history is desired, even as "add more commits in response to feedback and squash at the end" is probably a more natural way of working for many problems.
This means any solution that addresses the "stack of multiple commits" use case also needs to address the "amendments in response to code review" use case. You also need to be able to rebase a change onto an update version of the change it depends on.
This raises some important subtleties. My current mental model of this is that all/most of it's possible but some of it requires further thought to articulate a good workflow, and yeah, some things get undeniably clunkier even as others get easier. To sorta echo BDavis elsewhere, this thread is probably a good starting point for a "cookbook" style collection of workflow problems & solutions.
One of the subtleties this raises in particular is that you might have a chain of dependent commits that you intend to be merged as-is without squashing, but then you also need to make some tweaks to some of the commits in response to code review. Then you have a branch where some of the commits need to be squashed, but not all of them. This may well be resolvable by manually squashing the ones you want to squash, then merging the result, but the tooling would need to support that without causing too many problems (for example, it'd be really annoying if you needed to create a new change submission that's not/poorly linked to the original, or if diffs that show you what changed between iterations of the change are not available).
It does seem GitLab supports this sort of workflow by allowing either the submitter or the reviewer to choose to squash merge-requests on a per-merge-request basis.
That is, for some merge-requests there may be a logical order of commits and for others that may be less critical. The unit of change for the GitLab merge is the merge request as opposed to the patchset/changelist.
I'd also be curious about how well GitLab UIs scale. The most complex project I have been involved in was probably AuthManager, where the main commit went through nearly two hundred revisions, had hundreds of code review comment threads, and several dozen comments depended on it, often across repositories. Gerrit didn't handle that super well, but it was functional. Github, at the time, would have been crippling. (I understand they have added some features aimed at large code reviews since then.) Can we try recreating something like that in GitLab and see how it looks?
Another feature I rely on a lot is the ability to compare different patchsets (to see how a comment got resolved, for example). At least on the Github interface, I don't think either force pushing or multi-commit branches really support that (although of course you can construct the diff URL manually).
Yeah, I think in the GitHub/GitLab paradigm it is preferable to have multiple commits in the branch as feedback is addressed, in part so it's easier for reviewers to see how things got fixed. The problem is that if the idea is to squash when merging, there is more work added to the reviewer to write a new commit message that incorporates all of the messages in the branch.
In particular, Gerrit lets you version and review the commit message itself across multiple iterations of the commit, which is really useful both for correcting mistakes in the commit message and for the use case @KHarlan (WMF) mentions where the commit message evolves along with the contents of the change.
I went and did some quick experimenting on gitlab to see whether branching from existing merge request branches would work for the stack-of-commits case.
It sort of works! You have to strictly stick to merging the stack from the bottom up. I.e. if you have branches `master > A > B > C` you have to merge C, B, A in order. If you do that, it updates the merge request based on the branch you just merged into to include the commit you just merged (and its merge commit, which may get a bit noisy).
Drawbacks:
The fast-forward merge option is per-repo and is on the settings page (see screenshot at: http://tyler.zone/merge-method.png).
While the UI is different in Gerrit vs GitLab, the underlying mechanisms here are the same—that is, in Gerrit you are merging a ref (e.g., refs/changes/yy/xxxyy/1) into a branch (e.g., main) and in GitLab you merge a ref (refs/heads/fix/T1) — which is itself is a "branch" — into a branch (e.g., main).
The creation of those refs in Gerrit is automatic and we rarely think about it. Pushing via e.g., git push origin HEAD:refs/for/main causes Gerrit to make changesets for every commit from origin/main..HEAD. In GitLab making each ref has to happen locally rather than automatically. This can be accomplished via something like:
while read changeset; do
git push origin "$changeset":refs/heads/thcipriani/changes/"$changeset"
done < <(git log --format=%h @{u}..HEAD)
to create a branch for each change. It is noteworthy that this is fighting the "feature branch" strategy the drives merge-requests. That is, if all your changes are related (i.e., a feature branch) then, in a GitLab world, these become a single merge-request.
Merging from the bottom up isn't necessary for chains of merge-requests as far as I can tell. For example, let's say:
A, B, CmA, mB, and mCmA's merge target branch is mainmB's merge target is mAmC's merge target is mBFrom here, the logical thing might be to merge from the bottom up: merge mC -> mB; mB to mA; mA to main—this works; however, it's not the only thing you could do.
You could merge mA to main (which deletes the mA branch by default), and then change your merge-target for mB from mA to main. This is, in essence, a "rebase".
This has been a long-winded way of saying: all git workflows are possible in git.
In many ways Gerrit aligns more closely with how git works (branches weren't always things) and the abstraction it provides makes some workflows easier. The merge-request workflow is a different abstraction than Gerrit's patchset-based workflow and it requires users to make explicit what was once an implementation detail of the tooling: namely that there are refs that represent a set of changes that we're trying to merge to a branch. This will have an effect on how we write changes.
If GitLab's abstractions makes a certain workflows more onerous then we should document those workflows to see if we can work with GitLab/the GitLab community to make them easier.
> The fast-forward merge option is per-repo and is on the settings page
Per-repo for that setting makes it impossible to replicate the gerrit flow, I'd think? Because outright replicating it requires that we fast-forward the merge-into-another-merge, but then have a merge commit for the final merge into master. (I don't actually have an opinion about whether we should have that final merge commit or not, I'm just looking at differences.)
> This has been a long-winded way of saying: all git workflows are possible in git.
I do agree with this! However, I was trying to examine it from a standpoint of what Gitlab's built-in tools would allow without needing to drop down to the command line or manually intervening with the merge request settings after every merge.
I think about this more in terms of the goals, and why i submit a 5 patch chain instead of a 1k+ line patch to be merged. These patches are between 100 and 200 lines each. They are small and easy to review and think about. They do individual small things, and the sum of all the patches achieves my goal. I often write these initially as perhaps one or two patches, and then refactor the commits into individual patches for ease of review. Each patch in the chain is focused, in my experience this greatly simplifies review
The simplest example I can think of is a two patch chain, the first patch refactors what exists. This should have no functionality changes. This patch can stand on its own as an improvement even without the followup patch adding functionality (typically, at least). Especially when it comes to puppet i want to deploy that refactor without the additional functionality, as further proof that the refactor made no changes to the outputs, only to the structure. The patch adding functionality should be merged and deployed separately. This is in part about isolating the failure domain, so we know which bit of what broke things.
I don't need the gerrit workflow, but i do need a workflow that allows me to write 1k+ lines of code, and not force someone else to have to decide in an all-or-none approach if this is mergable. I need a way to write pieces of code that are related and build on top of the previous patch, but deploy them as individual units to isolate the causes of failure.
To add onto that: whether you end up merging one big change or several smaller ones also impacts what the history looks like. That's not just a cosmetic thing, it has real-life impacts on how useful tools like blame, bisect and revert are. In Erik's example, if the refactor change and the new functionality change are separate, and stay separate in the merged history, we'll be able to identify which of them caused a particular bug, and we won't need to do baby-with-the-bathwater reverts as often.
+1 to being able to keep the chain of small patches separate; when I'm looking back through puppet or MediaWiki core or other repos, small commits are easier to eyeball, search through the commit history, etc to find the source of a specific change or narrow down when a particular behavior changed. And especially in the case of commit messages, I look at the related patchset to be sure the code does what the change says, or to understand what is being said; that's much more onerous with a thousand line squashed commit.
There are CI implications as well. In this chain-of-small-patches approach, *each patch in the chain* is expected to pass CI. In the feature-branch approach, usually CI is only run on the end result. That's fine if you're going to squash all the commits at the end, but causes issues for git-bisect if you don't squash commits. (Take a look at the bisect alias here https://github.com/smarkets/marge-bot#some-handy-git-aliases for the complicated workaround to make git-bisect work as we're used to.)
From what I've seen in GitHub, each commit in a branch is run through CI. I assume the same could be done (is already the default?) for GitLab as well.
The way I've been thinking about the PR model as it applies to our current gerrit workflows is that each branch would be the "patchset chain" that we have in gerrit. For example, looking at this patch:
Then this patch which is in a different repository would have a merge request ("Use MobileFrontend overlay for language question") that depends on the merge request for "Add welcome survey language question"
In other words, rather than thinking of each patch in our existing chains as separate merge requests, I think it might be a bit more intuitive if they are separate units (commits) within a single merge request.
> From what I've seen in GitHub, each commit in a branch is run through CI
It is possibly configurable, but by default only the latest commit at the time of pushing gets tested. See this example where my first push was two commits, and so the first commit ("Fix documentation whitespace") was never tested.
Ah, it seems like it's not possible in GitLab, not yet anyway; there are a few issues and some WIP merge requests around it.
The task tracking that request is now at https://gitlab.com/gitlab-org/gitlab/-/issues/25148
FYI, GitLab PMs are known to review thumbs-up emojis (similar to votes) on a periodic basis to judge need/interest from the community.
Those following along here, please see the comment from Brennen on the primary consultation talk page (sorry for forking it...) on how to test out the workflow in our test GitLab instance. The consultation WG would greatly appreciate a few people to give it a go and giving feedback on how to best utilize GitLab workflow options to meet our needs. Thanks!
I'm a bit weary of using open core products after what we had with varnish (another open core product we have in production that's being phased out in favor of a fully open source one). Varnish worked well in scale but it was intentionally lacking important features (like TLS support) to encourage users to buy the full version. This is the open core business model and I'm afraid we are going to end up with a similar situation with gitlab. Even if it looks like we have everything we need for our usecase, there's no guarantee it'll be still accessible tomorrow (I'd rather we don't fork gitlab). Has this been taken into account?
This is one of the things I was (am) worried about. For the time being, GitLab has been moving in the opposite direction; that is, moving features from the enterprise edition (EE) to the community edition (CE) with some regularity (https://about.gitlab.com/blog/2020/03/30/new-features-to-core/).
The behavior of moving nice feature from EE to CE could change, of course, if the company decides without much notice to pivot -- company behavior is sometimes capricious, unfortunately.
I talked, briefly, with folks from KDE, Gnome, freedesktop.org, and Debian: there does seem to be a community of like-minded organizations that are working on and with GitLab and GitLab, Inc, respectively. The impression I have is that there is enough of a a critical mass of open projects that run the community edition to keep the community edition in a functional state.
For more, see https://about.gitlab.com/company/stewardship/, notably:
There's more there and it's worth a read.
I also want to re-emphasize the community of like-minded orgs using GitLab right now in the same way (self-hosted Community Edition); there is already positive examples of cross collaboration and support.
Thank you for responses. I feel it answers my concerns. Specially joining the group of open-source orgs using gitlab would increase the CE version bargaining power.
Possibly late to the party, but GitLab also have their open source deal, where Wikimedia can get much of the premium EE feature set for free. Requirements are non-profit org, and all the hosted code is public and open source licensed: https://about.gitlab.com/solutions/open-source/program/
We should be able to preserve the review comments and metadata as it's already stored in the repo in gerrit.
Would it be possible to migrate review comments from Gerrit into GitLab? I guess it would be kind of tricky, since you'd have to create closed merge requests based on patchsets in order to recreate the comments. And somehow also map commenters in Gerrit to users in GitLab. If it's not possible to migrate comments, it would be useful IMO to keep Gerrit in read-only mode so that past patchsets can be viewed. I've used past discussion on several year old patches to learn more about why something was done in a particular way, and it would be nice to preserve that ability.
That's actually a pretty good point. I as well use gerrit in that way quite often. It's very educating to figure out why something was implemented in the way that it was, and while some times the commit message carries that information, it also quite often doesn't and that information is in the reviewers' comments. So I would love it if we could keep that knowledge around. Note btw that maintaining a read-only Gerrit will invariably be more expensive down the road in terms of maintenance so it's better to investigate early if it can happen
We thought a bit about importing, but, as you mention, that's a can of worms. Currently leaning towards moving projects toward read-only as migration progresses and eventually moving to a static dump of Gerrit review content.
Importing seems tricky because of the completely different process and assumptions. Just like we didn’t really import anything from the old SVN-era Special:Code and just left it as-is in a read only version, I think the same would be ok with gerrit. Converting gerrit to a static set of pages seems like a good way to reduce the maintenance burden.
Gerrit exposes review comments as git notes and they can be downloaded locally. While that doesn't really replace a web interface for ease of use, it is searchable, which can sometimes come in handy. (E.g. you can use Gerrit labels to tell who would be a good reviewer for a given piece of code.) It would be nice if that could be preserved in some form, too.
Ah, I remember you mentioned that before but had forgotten it. (For those interested, there are some notes on usage in the Gerrit advanced usage page and this blog post has more useful information about how it works.)
So, I think that would satisfy the case for merged patches, but ideally we could also preserve patches (and comments on patches) that are not closed -- there appear to be +5,300 of those.
Gerrit stores everything in the git repo in some way (e.g. you can see the comments on a given changeset with a command like git fetch "https://gerrit.wikimedia.org/r/mediawiki/extensions/GrowthExperiments" refs/changes/12/609212/meta && git checkout FETCH_HEAD) so maybe that repo could be preserved independent of Gerrit.
It is possible to preserve refs/changes/*. In GitLab these refs wouldn't be exposed via the UI since the refs/changes namespace is a Gerrit concept, but those refs could be preserved in the repo.
Also, aside from directly looking at the repo history / blame, we have a lot of phab tickets containing links to gerrit patches, and we'd be dropping a lot of useful data if gerrit suddenly became inaccessible. Even just a separate static archive would still be a real pain if links broke without redirection.
Notably, the links from phab don't actually contain a useful reference to the commit, so if I see "Change 588935 merged by jenkins-bot" and the subject line of the commit, that's not an easy thing to map to the aforementioned git notes.
There are a few steps to map a change to review-notes:
refs/changes/35/588935/meta is the discussion about change 588935 (https://gerrit.wikimedia.org/g/mediawiki/extensions/DiscussionTools/+/refs/changes/35/588935/meta).refs/changes/35/588935/5 and find the review notes from there.Please see GitLab consultation!
Is there going to be an actual consultation/RfC/gathering of requirements from the development community as to what is wanted out of a code review platform, and not just CI? Or did this already happen and I just missed it? The only thing I'm aware of is T167547 which was declined.
FWIW most of the feedback on Talk:Wikimedia Release Engineering Team/CI Futures WG/Requirements seems to have gone unanswered, so it's not clear to me exactly how to contribute to this process.
There won’t be an official RFC (or RFC-like) process for this. The Eng Productivity team has been working on CI planning for some time now and more recently taken up the question of code review based on feedback. After discussing with the Architecture team lead, we decided that an RFC process wasn’t the right choice.
We welcome, expect, and encourage feedback and recommendations along the way as we move to GitLab so that we can address as many needs as possible. That’s me saying: no one tool will address everyone’s needs and also we shouldn’t expect a complete 1:1 parity between GitLab and Gerrit.
Just to be clear, you're saying that a firm decision has already been made, and there's no turning back now?
If so, that's incredibly frustrating to hear, that a decision has already been made, in private, without consulting other technical community members. It gives the feeling that our opinions and values are not important enough to even justify asking for them, let alone incorporating them. I think many of us would've been excited about GitLab, but right now this process is leaving a sour taste in my mouth.
It's hard for me to now justify the time I've spent developing Gerrit-specific tooling over the past few months for people to use when it was clear that in the long run it would be worthless.
Are there any guidelines on when an RFC-like process is the right choice? If so, are they public?
Naively I would think replacing one of the two essential tools all developers rely on is exactly the kind of thing where gathering feedback first would be the right choice. So this leaves me a bit worried about what else is considered now to be better handled by a WMF-appointed Director of Something deciding it alone, without even trying to understand how users will be affected by the decision, much less giving them a voice.
Wikimedia_Technical_Committee/Charter#Areas_within_scope says if it's one of these three, it falls under TechComm: "Strategic, or Cross-cutting or Hard to undo". The word "Strategic" is vague so no opinion on that but it's definitely cross-cutting AND hard to undo.
TechComm is being phased out, with the introduction of the Architecture team. I'm not sure there will be an RFC process going forward, at least not in the format we're used to.
Which I'll note is the polar opposite of what has been accepted in the movement strategy (See Coordinate Across Stakeholders: "Create a Technology Council to establish processes for introducing new functionalities onto Wikimedia platforms and tools. The aim is to improve communication between staff developers and other technical contributors to better network, coordinate innovation, provide and obtain support, and foster input on decisions and resource allocations that impact communities."; and the older and less mature but more detailed Evaluate and Decentralize Technology Components: "Set-up an iterative strategic consultation in which long-term goals and visions can be determined in a participatory manner. This process should lead to a maintained public roadmap with project dependencies outlined. From the roadmap affiliates and volunteers can make proposals to build components of the roadmap. These proposals will be vetted for their technical merit, cohesiveness with the technology architecture through the “Technology Council”.") But presumably there will be a better place to discuss that when that decision gets announced.
It should be clear that an architecture process not involving volunteer developers and open discussions can not work in this community. Being open to input from anyone makes reaching the optimal solution possible.
TechCom has worked out very well so far, and I see no reason to replace it. Why do we have to reiterate such situations again and again, with the WMF obviously not learning anything along the way? It can not be so hard to be so hard to understand which kind of decision-making usually leads to a huge public outcry and to just not do it this way. Unless, of course, it is happening with intent.
@Tgr thank you for assuming good intent. As you know I was involved in the movement strategy and my intention is not to consolidate decision making into the Architecture Team. I do seek to improve our decision making processes by making sure we clarify who the stakeholders are up front and what the problem is we seek to solve prior to discussing solutions. While a move to GitLab is a big decision it is not a system architecture decision in my view. I also don't think it falls under the TechCom charter.
@MGChecker There is no intention to not involve volunteers, nor only have closed discussions.
@KChapman (WMF) I apologize, that comment was unfair, given that you are the main author of one of the aforementioned recommendations. I still think it was unfortunate to make that kind of decision without any kind of public feedback. From within the organization it might look like just another restructuring, but for the technical community, TechCom has been its core governance structure, and despite technically being selected by the WMF, in practice it has been a fairly meritocratic one, with an open application process and whatnot. Replacing it with something regulated by opaque HR decisions is going to raise eyebrows; quite possibly just due to lack of information, and if we understood the planned role of the Architecture team, we'd be perfectly happy (there is certainly a need for better resourced and better connected architectural planning! We even had a draft recommendation about it), but there's an issue here with communication, at least.
Anyway, looking forward to hearing more about that plan and maybe discussing it at some other place and time.
@KChapman (WMF): I would personally consider the people using the software the most (which includes the volunteers) as a very important group of stakeholders. Being blindsided by what appears to be a decision that has already been made without any prior consultation leaves a very sour taste. The sad part is that, if the WMF did bother to surface this before making a decision, I'm pretty sure there would be overwhelming community support in moving away from Gerrit and into GitLab.
The initial announcement however made clear that a decision was already made behind closed-doors, with no statement that the community could modify the decision, only that we'd be consulted for "feedback and recommendations" that is likely to be ignored if it doesn't already fit the "we're moving to GitLab" decision. While it is true that this is explicitly out of scope of the TechCom charter, surfacing the idea here on MediaWiki.org for community involvement before making a decision that completely upends the workflows of every volunteer developer still would have been a smart move.
However, it's not too late to start bringing the planning into the community. I strongly suggest that the WMF creates an informal RfC here on MediaWiki.org ("informal" in this case meaning "not under TechCom's purview"). Even if the WMF views decisions made as a result of the RfC as advisory instead of binding (although I'd hope they'd be binding), a move to a completely different code review platform brings in a lot of complications, and the earlier you can get community feedback on what sorts of workflows and use cases need to be supported, the better everyone can plan for a migration.
As someone else mentioned in a previous comment, having a backout plan in the event GitLab's code review features prove to be untenable should also be part of the planning (meaning, we'd also want to investigate ways to integrate Gerrit with GitLab CI).
@Skizzerz I'm not sure if "you" in this case means me personally, I wanted to be sure there wasn't an expectation of action on my part. I wanted to clarify that the GitLab discussion is outside of my scope. If "you" means the WMF then fair enough.
Sorry, I did mean "you" to mean the WMF. I thought I caught all instances of that when copyediting before posting it originally, but I guess I missed some. I've edited the comment to reflect the intent :)
My call to action is for the WMF as an organization to begin an informal RFC so that we can start the process (better late than never). Specifically who within the WMF champions that is not my call to make. If there is something you personally can do, it would be encouraging the appropriate people to make that happen.
The fact that the TechCom is being phased out in favor of Architecture team is quite concerning to me and I have no venue to give the feedback (and I assume it's already decided behind closed doors since this is the first time I hear about it...). For example, I read Wikimedia Architecture Team#Decision making to understand how gitlab is decided and this is the text:
Conway’s Law is truth: The way we structure communication as an organization inherently designs our solutions and the system as a whole. To be effective, we are strongly focused on improving the way we "architect" solutions by improving the structure of solutioning together. Beginning with structuring recommendations that are more integrative and tolerant of ambiguity, while still actionable and quickly iterative. We are actively working with engineering and product to understand how best to support organizational decision making without inadvertently rebuilding the same structures that created our legacy system. We are also creating a “system view” framework in which people can position decisions and connect them to other decisions in a more meaningful way.
This is too general and vague to be of any use. You can put this as "decision making process" of every other team at WMF and you would see it applies there too. Compare it with Requests for comment that is extremely detailed and well defined and went through several iterations already (the last one was quite recent). It seems we are throwing lots of years of work with this.
Well, maybe it's being phased out, but so far, it's a MediaWiki.org policy. I think we should follow policies which are currently available at MediaWiki.org, because that's what people uninvolved in changes will expect to be followed.
Per Wikimedia Technical Committee/Charter#Areas within scope (emphasis mine), "The scope of the committee includes all the official software that serves Wikimedia users. That includes MediaWiki and extensions, services, and other software running on the production cluster ... The scope also does not include developer tools ... or other non-production technologies."
So Gerrit (and in general developer productivity infrastructure) changes are clearly outside TechCom scope.
Hi Grant, welcome to Wikimedia. I see this is your first time using this account. Who or what made you think that making such a huge decision on your own without community consultation was a good plan? You might have consulted other people, but you're the top of the corporate chain, so in the end, it's your decision.
It shows severe lack of awareness of how things work around here. Are you even aware of the big community uproar around the renaming? You seem to be acting like you are the CTO of some kind of corporate entity. Wikimedia isn't. Big changes are usually discussed with community members to reach consensus. Just announcing a change like this is a slap in the face for members of the community.
Applying Hanlon's razor here: I'm assume you weren't aware of this. So please reconsider and act on it now.
Who is the "Architecture team lead"? Is it Kate as being the Director per Wikimedia Architecture Team?
Previous significant development tooling changes did go through an RFC or RFC-like process:
There are also two previous RFCs for migrating away from Gerrit:
Situation may of course have changed, but I see no reason to skip the consultation process to determine whether this is really the best course of action.
This should certainly be an RFC and go through the formal process the moment it touches the wider MediaWiki development community. Swapping just CI out would be an infrastructure change that doesn't modify the workflows that the vast majority of contributors are using (as the vast majority of contributors are just publishing and reviewing patches, and the CI component is largely just a "black box"). Changing out the Code Review platform entirely however must go to the wider community for feedback before any changes are implemented. Doing otherwise is equivalent to the WMF saying "fuck you" to the community of developers and contributors by depriving them of any meaningful voice and will likely drive away contributors even if GitLab ends up being a better platform for that reason alone.
Another voice urging reconsideration of the decision not to RFC this. Code review tools exist to serve the needs of contributors and reviewers, be them WMF staff, affiliates, or volunteers. The RFC process exists to ensure drastic decisions with wide-ranging implications like this one are actually in the best interests of everyone involved. Making a decision like this without any outside review is quite daft and downright dangerous.
Even as someone who really really hates gerrit, I'm rather disappointed i'm hearing about such a wide reaching change second hand, after the decision has already been made.
I would like to echo what Bawolff and others are saying. Quite disappointed about the process. This process seems very similar to what is currently happening at meta:Community open letter on renaming.
What in the world ? I mean I’m first in line to switch to Gitlab, I even chose it over phabricator back when we picked that, but this is very unexpected... I’d like to hear more
Seconding what has been said here. As the person who hated gerrit so much that wrote a bot to mirror PRs in Gtihub to gerrit six years ago for pywikibot, I'd really appreciate following the proper formal process for this. We have an ARB for a reason.
Plus one. If anything needs a RfC, it's definitely changing one of core developer tools. We are a community, not an authoritarian group with an all-mighty dictator.
I'd like to echo the feelings of @Legoktm and many others above regarding this unconsulted migration. I feel very dissapointed. MediaWiki and its extensions are developed by many volunteers that deserve being asked before taking such drastic decisions. Please reconsider and open a RfC.
I am echoing what everyone else has already said. The code review tool is one of the most important tools used in an environment like this. Please do not completely change it without asking everyone what they think about the change.
I'm personally very strongly in favour of a switch to GitLab; as a new contributor, Gerrit is alienating and confusing, and that's not even mentioning the trouble inherent with git-review and its "pretend to be part of Git but actually don't really be" hacks. That being said, I agree that it's important that this is a community decision - the MediaWiki codebase is not owned by any one single person or team.
Hi All,
Responding to several questions and asks around this instead of replying directly to each.
It is certainly not our intent for this to be a decision made without consultation. I very much expect the teams involved here to consult with the community on how this gets implemented in terms of critical technical issues and developer workflows, as is already happening on other threads on this talk page. As for the choice of the tool itself, I believe the Engineering Productivity team did their homework in terms of recognizing a trend in our developer satisfaction survey that pointed at an underlying issue with code review that needs addressing and worked through a series of options to arrive at the recommendation they made to me for us to move forward on Gitlab.
As for the broader decision itself, I was trying to balance out moving forward on a decision that is never going to please everyone with a need to work towards solving very real concerns in both our continuous integration and code review tools that Greg and team have outlined on this page, knowing full well that us developers get very used to their version control and code review workflows. My overall sense is this is not an area we as a community and staff should be maintaining one-off solutions in, as it reduces all developer's effectiveness and makes our community less welcoming, even if it is very much a set of tools people are used to. In other words, my hope was that by picking a tool that is well accepted in the broader open source community, is in line with our values as an open source project, and has standard developer workflows that *most* developers are familiar with, we'd be able to please most people, even if there is some relatively short term pain (in the grand arc of what we are all trying to do) for folks on their particular toolchains.
As @Tgr cited above, my impression was that we didn't need to do an RFC for this kind of decision, even if sometimes in the past it has been done (or not, as seems to be the case too).
Grant, I'm sorry, I fully realise that what I'm writing is inflammatory, but there's really no other way to get this point across. Is WMF never going to learn from its mistakes of pushing its decisions down users' throats?
Hey @MaxSem,
No worries, I get it. I've been in open source as a volunteer myself for a long time. I'm not trying to shove this down anyone's throat. This is an important part of people's workflows, as a dev, I get that. I'm reviewing with the team how to incorporate the feedback here. I thought the main consultation part, to take place over the next year, would be around how to port the workflows and make sure people's needs are met, but clearly we need to step back one layer on that.
I'm extremely worried by this trend of WMF management not to consult their customers anymore. I'm seeing it all over the place recently. This isn't Facebook or Google. Get out of the SF bubble.
I'm sorry if this comes across as unsensitive, but as a long term volunteer contributor and arguable one of the people who tends to stick up for WMF in our communities, when there are actual problems that WMF needs to deal with, you are making it VERY hard on me/us to defend WMF when it creates transparency and equity problems like this where they simply DO NOT need to exist.
And I'm in FAVOR of gitlab and I feel that way.
Hey @TheDJ,
Apologies, it is my intent to consult the community. I thought that would take place as it relates to the porting of code review workflows and wasn't necessarily needed on the choice of the tool itself, but clearly we need to step back one layer based on this feedback.
FWIW, I don't live in SF and I'm thankful every day this isn't either of those two companies you mentioned. :-)
Sorry, I don't think you answered the initial questions I had.
I have plenty of more questions, but right now I'm having difficulty even fathoming why the mere existence of this process had to be kept secret, it's just not what we do.
The Wikimedia technical spaces are made up of a different number of communities, sure it's dominated by MediaWiki, but there's Toolforge users, Pywikibot developers and so on. Then there's all the projects that have gone to off-site (usually proprietary) hosting sites, like the Commons app, OpenZIM, various tool developers, UTRS, and so forth. GitLab should have been an opportunity to reach out to those communities, figure out what they're missing and see how to bring them back onto our shared, free code hosting and review platform for a stronger and more unified Wikimedia technical community.
Instead, it's just being fractured even further.
Hey @Legoktm,
I'm reviewing based on the feedback here. I don't believe there was some intent to keep this secret so much as we thought the main consultation would be around porting of workflows, but I get it can feel that way. I do agree there is a lot of splintering (which is both a strength and a weakness of open source) and I would like for us to be able to reach out to those communities with tooling that addresses the reasons they felt compelled to leave.
I'm not really interested in peoples' intents because then that leads down the path of the blame game, which I don't want to play. I'm more interested in identifying and fixing the process problem, which is what will actually prevent this from happening again.
As I see it, there should have been a notification/announcement/etc. that the scope of the CI refactoring project was expanding to include code review. That would have triggered a broader notification that moving to a new code review platform was on the table and solicited wider input, and so forth.
Hi Grant, thank you for engaging with the concerns here!
About the naming, RfC can (somwhat unhelpfully) mean either the TechCom process or feedback gathering and discussion in general. Up until about 2015, we had an unstructured RfC process for large-scale decisionmaking, which was used for both architectural and non-architectural decisionmaking, including things such as switching from Bugzilla to Phabricator and switching from Gerrit to Phabricator (eventually rejected, after the first few experiments turned out not so impressive while Gerrit improved). (I don't think there was any kind of explicit process for switching from SVN / Extension:CodeReview to Git / Gerrit, the third past change comparable to the one we are discussing now, but it was preceded by massive amounts of mailing list discussion.) After ArchCom / TechCom was established, it took over managing RfCs related to system architecture, which in practice is almost all of them; we had a separate Phabricator board for TechCom RfCs and generic RfCs (the latter was more recently renamed to Proposals to reduce confusion).
So I don't think most people talking here about an RfC mean there should have been a TechCom RfC, but that there should have been a participative decisionmaking process (the Foundation has recently started calling these consultations, in part to avoid the different and very specific connotations the term "RfC" has on content wikis). Both because we have an intricate and diverse tech ecosystem with many different niches with different requirements, workflow needs and contributor backgrounds, and the information needed to understanding that environment is distributed in many heads, and we need to bring it together to make the right decision; and because that's what aligns with the culture and values of our movement, and open source in general. The Foundation's guiding principles say that the WMF should share decision-making power with the community; the movement strategy principles likewise ask for participatory decision-making. More generally, and at the risk of being mistaken for a Marxist :), open source is one of the very few special environments where workers can truly own their means of production, and excluding them from that ownership will alienate them from their work. Fundamental changes to your work environment happening over your head are disempowering and disaffecting, even if they are otherwise reasonable changes. (It doesn't help that this is happening in the context of the Foundation having announced other, non-technical fundamental changes, which are far more controversial. So tensions were high anyway.)
I find the current course all the more unfortunate because I don't think there's all that much difference between a consultation and a precommitted decision in terms of what specific actions will be taken. A consultation would include gathering feedback on how people used Gerrit (or whatever else they were using), pain points and high points, things they would like to see improved and things they would like to be preserved. It would gather feedback from people with experience with GitLab. It would result in creating some kind of feature comparison matrix, with a summary of what we would gain and how we'd minimize losses. It would set up a trial GitLab instance and migrate some projects that volunteer for it, to verify that our understanding matches reality. These are all things that will mostly have to be done anyway! So IMO the best course of action would be to un-commit to a specific outcome, set up a working group (ideally with some volunteer participation) to do the work of surveying, requirements gathering, feature comparison and testing, with various feedback opportunities along the way; and have them make a go/no-go decision at the end of that process. Again, I think most of those steps would be necessary in some form anyway, so we'd get a more aligned and empowering process almost for free.
Also, I would like to caution against interpreting things in the narrative of old vs. new users. (More as a precaution, not because of anything specific you have said.) On content wikis it is an unfortunately common pattern that established users prioritize their own productivity above the ease of use of the software by newly joined editors, and the core community gets into fights with the WMF which considers community growth more important in the long term than power-user efficiency.
That kind of conflict is almost nonexistent in the technical community. Most people here care a lot about engaging new developers; all the feedback we got about the related movement strategy recommendation was positive; pretty much everyone commenting on this page has spent many volunteer hours supporting new developers in one capacity or another. To the extent people are skeptical about replacing Gerrit with GitLab (which might well be a small extent, if we put process concerns aside), it's because they are doubtful about it being a significant improvement to the openness and inclusiveness of our community, not because they don't care about that aspect.
More specifically: I think there is a widespread consensus that the main problem with our onboarding and retention of new developers is code review. As in, not code review tooling, but someone actually spending time on reviewing their code. Patches go unanswered for months or years (or, often, forever) because of insufficient capacity for code review; that capacity is by far our worst bottleneck for developer engagement. A tooling change that reduces that capacity will be a net negative; it doesn't matter whether it will be easier for newcomers to contribute patches or not, if there's no one to merge it. Becoming serious about code review would be a painful decision for the Foundation, because all that review time will have to come from somewhere, and thus result in less time for the Foundation's own programmatic goals; but in my opinion, it is a necessary one. I'm somewhat skeptical about attempts to make our community more welcoming which don't try to address this fundamental problem.
(As a corollary to that, I'd really like to see GitLab evaluated specifically for its ability to support code review of volunteer patches. Gerrit has many weaknesses, but one thing it is pretty good at is searching and dashboarding across many projects, useful for keeping track of volunteers patches needing code review or guidance in areas of code with unclear ownership, which is most of our code. Github, with its sweet spot of small unrelated projects with a small group of maintainers, used to be almost unusable in that regard, although I haven't used it recently so maybe they are better now.)
In the most recent developer satisfaction survey 77% marked code review workflows as a 5/5 (and another 18.7% gave it a 4/5) for how important it is to their work. In terms of the "glue" that holds the developer community together, code review (along with phabricator, mailing lists, IRC, and in person or virtual hackathons) is one of the main places where we talk to each other, critique ideas, improve things, etc. It's a key social space as much as a technical one, is what I'm trying to say.
So, while understanding that good intent is behind this move (the desire to improve weaknesses in the current code review workflows, and to do so quickly), I think it is understandable that folks are upset that something so ingrained in day-to-day workflows is being replaced without asking for consent or feedback.
With regards to whether an RFC is required for this or not, I guess I would say that just because something can be done in a certain way doesn't mean it should be. Even without a formal RFC, something like "hey, after studying this and for X, Y, Z reasons, we are proposing GitLab, we'd like to hear if the are any blocking concerns, these are open questions we would like your feedback on. Unless there are blocking concerns we will start the migration work in N weeks" would have been IMO much better received. I assume this project will take months if not years; would a small consultation period really have hurt? It feels like this move could have been a nice "win" for our developer community but instead we are expending time & energy talking about process failures, and making people feel as though the things that they themselves have ranked as being of highest importance to them, are not worth consulting with them about.
Anyway, sorry if the above sounds too negative; I really appreciate the hard work that everyone has put into our code review and CI infrastructure more generally. I hope the outcome from this discussion is that we find a way that folks can feel heard, engaged and invested in the success of the proposed changes to our code review system.
I echo this. It's important for us all to remember that real people have spent a significant amount of time working on this, trying to find a good solution, and they shouldn't feel like their efforts have been in vain, or that people are trying to say that what they did isn't worthwhile. It's simply a question of trying to make decisions together, rather than in silos.
Hi All,
I appreciate the feedback and the many thoughtful comments. As many have pointed out, this is a long process and we don't need to rush the Code Review side of this, even if I know the teams involved are anxious to have clarity in what their upcoming work looks like. I thought we were doing a consultation as part of working through what the migrations of workflows would look like and how it will affect all involved, but after reviewing the comments here, it feels like we should step back and be sure to consult one level up on the choice itself.
Ideally, I'd like to give the team the go ahead to use Gitlab for CI, as that is a behind the scenes implementation detail and our current solution is not viable (Python 2.x EOL, Zuul upgrade needs.) At the same time, though, I worry that if we do that a Code Review consultation will feel like it is then biased towards Gitlab or already the choice. I'm asking @Greg (WMF) to take the lead on setting up a working group to address the concerns here and they will commit to gathering the feedback on code review and reporting out. We will pause on the code review decision for now.
I agree that the CI part can move forwards as that is a backend change and won't result in any changes to existing developer workflows for the majority of developers.
One note I'll bring up about the CI migration: can we look into making a "default" profile that runs on all repositories, or at least all repositories that match a particular naming pattern (such as extension-*)? Right now if we onboard a new extension, we need to also remember to manually make a patch against Zuul's config to enable CI for the new extension repository. It would be nice if new extension repositories were automatically opted into CI, and only those that need special configurations or to opt out would need to submit config changes.
I think you're somewhat underestimating the impact of switching CI platforms would have on developers. If you look at the requirements, a big one is self-serve CI, which does significantly impact how people work with CI.
(As for your feature request...I remember filing something similar but can't find it offhand. But addressing that is hopefully going to be done in the new CI system.)
No, I don't think that's appropriate yet. My understanding is that GitLab CI was chosen on the merits that it provides a better experience when integrated along with GitLab. On just CI platforms alone, Argo was chosen.
Switching to GitLab CI now, presupposes that we're switching code review to GitLab, otherwise we're picking the inferior solution.
If maintaining Zuul until the code review+CI consultation is completed is so untenable that it can't wait, then code review platform switching shouldn't be part of that process and should be evaluated independently (as it originally was).
Intuitively, Argo should also be much more straightforward to implement. GitLab is much larger and contains all sorts of things; big projects which have tried migrating to it seriously have reported all sorts of unexpected friction (things like it was surprisingly hard to replace the GitLab.com logo everywhere, reliance on Google ReCaptcha etc.).
I'll let the Eng Prod folks who evaluated weigh in, but I don't think Gitlab is an inferior solution even on CI. My review of the analysis put those two (Argo and Gitlab) neck and neck. Gitlab for CI I don't think needs to presuppose Gitlab for Code Review, even if it is an integrated solution.
@GIngersoll (WMF) and @Greg (WMF) thank you for responding to the concerns here and broadening the scope of the consultation with the developer community, I really appreciate it.
Thanks Grant. I look forward to having this discussion publicly.
I'll reach out to a few additional people/representatives from the volunteer/non-Foundation community to fill out the current Working Group and will treat that group as "Consulted" in the RACI sense. For the nebulous "newbie" group of users (aka: those who have not yet engaged with Wikimedia projects) I'll be using the expertise of our Technical Engagement team. I will provide updates to the public as we work (at some sub-page of this GitLab page, to be created), so in essence "everyone" is in the Informed category.
As we go through this discussion I expect the conversation to continue here on more focused workflow questions (best practices etc) in a two-way manner with the working group.
Thank you to everyone for your feedback thus far.
Thanks Greg. I'd suggest having at least one person from: the Pywikibot community, given that it was the last straggler in the SVN -> Git migration and have had real trouble having our needs met by the current CI system; and the Toolforge community, which has mostly not been successful in using Gerrit, instead hosting most code on Diffusion or external sites like GitHub and Bitbucket.
(I'm a member of both communities, so that's what I'm most familiar with but also my bias.)
This might be also a good opportunity to involve communities who have not used our CR/CI systems so far and see if a switch would make them more excited about it. Semantic MediaWiki would be the big one.
SMW was using our CR/CI until a few years ago... https://gerrit.wikimedia.org/g/mediawiki/extensions/SemanticMediaWiki
Not at this time, other than one that a team member installed on a personal server for quick testing.
A very constructive way that folks could help with this migration would be collect various "typical" pull request workflows and document their implementation as well as pros and cons. Gerrit is highly opinionated in how to use git for code review. GitLab/GitHub provides a lot more flexibility, which can be nice, but can also be a burden. If we end up with 200 repos and 50 different workflows that will in itself be a large impediment to some of the hoped for gains from moving to a more widely used git hosting and code review platform.
I think in an ideal world we would end up with consensus on a "preferred" PR based workflow that projects would be encouraged to adopt. Forcing a single workflow across all projects is itself not ideal, but having a standard process will make it much easier to explain how project/repo X differs from that Wikimedia standard practice and why.
I like this, @BDavis (WMF). I tend to look at these things as opt for the default for simplification of development, onboarding, documentation, etc. unless there is truly something unique about a particular repo. PR workflows are not an area of innovation for us, so let's try to let others trod the path before us.
+1 to having a standard workflow.
Would we be using a forking workflow (e.g. there is a canonical mediawiki/core repo, and my fork is kostajh/core, so then I submit merge requests from my fork) or their recommended approach that developers have write access to push branches to a single repo?
Then we should also talk about whether we would use their recommended GitLab flow (which seems kind of complicated) or Github Flow which seems to track what we currently do in Gerrit, e.g. there is a master branch and then patchsets ("feature branches") are merged into that, and master is what is deployed. I guess some of this depends on what the future holds for continuous integration / continuous deployment workflows.
In Gerrit, collaborating on a patch is kind of awkward. AFAICT, you can either push your change directly to the patchset, which is complicated because maybe the first developer doesn't agree with what you've done. Or you can push a patch on top of theirs, but then it doesn't actually get merged in until everything else in the chain does, so it is kind of awkward. Or you can leave inline comments with suggestions which is cumbersome.
In GitLab, I noticed they have a really cool feature where you can generate an inline suggestion that the user who created the merge request can then decide to accept or decline. I don't know what that looks like in terms of git history, authorship, and workflows (or if it is supported in GitLab CE), but I did want to raise it as a something that would be really nice to support as part of whatever standard workflow we adopt.
Repeating here what I have raised elsewhere: I find the timing of this decision unfortunate in that we have just upgraded Gerrit from a three year old version to a bleeding edge one, which brought major improvements such as hiding machine noise from code review discussions, or the ability of CI tools to leave inline comments. Satisfaction surveys based on people's experience with the old Gerrit version (which has been end-of-lifed a year ago, and which was mostly created before Google started stuffing UX desingers into the Gerrit team) are not necessarily indicative of how user-friendly Gerrit is now.
To be fair, none of the comments in the survey were about the things you mentioned (eg: inline comments or hiding machine noise) and more about generalized things like the workflow and the interface (which really hasn't change drastically). To put it in a different way: I don't believe anyone who choose to not use Gerrit will now move back to Gerrit due to this upgrade.
But point taken re timing.
The Release Engineering and SRE teams.
Hello,
who will be maintaining gitlabs at wikimedia (including contributing features or submitting stuff to the code base)?
Looking at the URL of this talk page, I'd say the Wikimedia Release Engineering Team. :)
The Release Engineering and SRE Service Ops teams. I'm going to leave it at the team level because we hope to reduce the "this person needs to answer all questions about it" issue.
If we move to GitLab... What do we do about the repos with their canonicals on GitHub? Should those move, because GitLab fixes many of the workflow problems and barrier to entry issues that people have with Gerrit?
Which seems to go with the ideas below of simplifying/normalising workflows too...
I think we should still mirror to GitHub
When it comes to documentation for onboarding new developers, I'd appreciate if we decreased the number of systems where people are supposed to clone a code repository from and create and share proposed patches for it. See also phab:T109939 about being able to identify which repositories are canonically hosted on Github (because that currently also makes our development statistics less complete).
My personal vote would be to move all canonical git hosting + code review to Gitlab. But I would also keep read-only mirrors of everything on GitHub for the reason I mentioned below, volunteer (and staff) portfolios.
+1 on continuing to mirror to GitHub. My reason for that is simply one of engagement. For better or worse, GitHub has become *the* social network for source available software development. Part of what participating in Wikimedia FOSS projects provides to volunteers is a portfolio of their work that can be shared with potential employers. GitHub is the most asked for way to share such things. If we take that incentive away, we will still have technical volunteers but we will also have a less compelling story to attract and retain them.
If I understand the pricing page correctly, push mirroring is only included in paid plans.
That is not correct. The pricing page lists, "Remote repository pull mirroring" as the only mention of "mirror". The local instance I have seems to confirm this. Push mirroring is in core, pull mirroring is not. This is exactly the same as Gerrit.
The Wikimedia Release Engineering Team/GitLab/Features table says "We evaluate code quality in CI at present, and would continue doing so."
But I think we could investigate the built-in GitLab integration with Code Climate, especially as there is already an integration that wraps around sonar-scanner; maintaining our own tooling takes time and effort.