Talk:Download from Git/Flow

Previous page history was archived for backup purposes at Talk:Download from Git/LQT Archive 1 on 2015-07-10.

I manage a MediaWiki site that is basically the REL1_39 branch plus LocalSettings.php and a few extensions I wrote myself. What is the best practice for maintaining this kind of set up with Git, so I can easily update the core MediaWiki software without disturbing my custom files?

I thought maybe I could create a local Git branch named mine that's based on origin/REL1_39 and add my files. I would never commit this branch to Gerrit -- it just stays on my local host. Then, when MediaWiki is updated to (say) 1.40, I'd just git merge origin/REL1_40 into my mine branch. But when I tried this locally, I got many merge conflicts.

Or, I could maintain my private files (LocalSettings.php etc.) in a separate local Git repository, and then at deployment time, combine the MediaWiki core files and my repository on our production server. But writing scripts to deploy files from 2 separate Git repositories feels a little hacky.

Is there a better way? Thank you very much.

If you got merge conflicts when doing git merge origin/REL1_40, it means that your own modifications touched parts of the code that changed between 1.39 and 1.40. I don’t think there’s an automated way to resolve those conflicts, no matter if you use a different branch, a different repo or something else – programs can’t read your mind to figure out what you wanted to do. A branch in the same repo is IMO the best solution exactly because of the conflicts: the conflicts highlight what cannot be merged automatically (and do so well before deployment, so the manual merging doesn’t cause downtime).

However, if you touched only LocalSettings.php and added extensions, there should be no conflicts: LocalSettings.php is ignored in Git, and extensions have their own directories. (Unless you happened to have chosen names that haven’t been in 1.39 but are in 1.40: in that case, simply rename your own extension.) So the question is: what else did you modify in MediaWiki core, and could those modifications either be submitted to Gerrit so that you don’t have conflicts in the future, or be reverted because the official version of MediaWiki now provides you what you need?

Thanks @Tacsipacsi! I actually didn't modify anything in MediaWiki core. Here's what happens with a simple merge of the kind I described.

$ git clone https://gerrit.wikimedia.org/r/mediawiki/core.git

$ cd core

$ git switch REL1_39

$ git switch -c foobar

$ git merge origin/REL1_40

This give me tons of merge conflicts.

CONFLICT (add/add): Merge conflict in .gitmodules

CONFLICT (content): Merge conflict in autoload.php

CONFLICT (content): Merge conflict in composer.json

etc.

What am I doing wrong?

I see. I didn’t realize what you effectively try to do is merging 1.40.3 into 1.39.7. Since they have independent history since 1.39 got branched, conflicts are actually not surprising. If you indeed don’t touch anything except for LocalSettings.php and your custom extensions, maybe the best solution is not to maintain your own branch of MediaWiki:

• Maintain extensions separately, one repo per extension. This is also how extensions are maintained on Gerrit.
• Maintain LocalSettings.php in yet another repo.
• Create a deployment script that deploys the extensions (this could go in the same repo as LocalSettings.php). Two possibilities:
  • Simply cd extensions && git clone ../../custom-extensions/extension-name
  • git submodule add ../../custom-extensions/extension-name extensions/extension-name && git submodule update (this helps with later updates, but makes the working tree dirty)
• Make sure LocalSettings.php is kept up to date, for example by copying it with the above script, or by symlinking it.

P.S. Please note that I’m just one person. This is how I’d do it, but it may not be the best solution.

Thank you @Tacsipacsi. I'll give that a try.

According to Download from Git#Keeping up to date, to do a minor update from, say, MediaWiki 1.39.1 to 1.39.3, I should do the following:

git pull
composer update --no-dev
php maintenance/update.php

This I do, and all goes well. However, after a successful update, I'd expect that running git status should yield an empty report (no unstaged changes, no untracked files), but it doesn't. Instead, I see all bundled extensions listed as unstaged changes, and all non-bundled extensions listed as untracked.

The first list I usually resolve by running:

git submodule update --init --recursive

Should this be added to Download from Git#Keeping up to date?

And what about the non-bundled extensions that appear listed as untracked? Is this expected? Shouldn't the extensions directory be included in .gitignore or something? What's the proper way to do a minor update?

It depends a little bit on how you're using git. If you are following tags, you would need to change to the 1.39.3 tag first. If you are following a release branch, then git pull would work.

I would recommend git pull --recurse-submodules if using a release branch so it automatically updates submodules.

Extra extensions you added being untracked is to be expected. Perhaps extensions other then bundled ones should be added to .gitignore, but that is not currently done at this time.

Ain't 1.31.15 the legacy and 1.35.3 the lts - exactly the opposite of what's said in the section?

Fixed. 1.35.3 is both the "LTS" and the "legacy" release. 1.31.15 is the "legacy LTS".

This isn't fixed, it's just differently broken. It now renders as "Currently, these are 1.36.1 (stable), 1.35.3 (LTS) and 1.35.3 (legacy)."

We can "fix" it temporarily by making it:

Currently, these are <tvar name=1>{{MW stable release number}}</tvar> (stable), <tvar name=2>{{MW lts release number}}</tvar> ([[LTS]]) and <tvar name=3>{{MW legacy lts release number}}</tvar> (legacy LTS).

… and then when 1.31 goes EOL changing it to:

Currently, these are <tvar name=1>{{MW stable release number}}</tvar> (stable) and <tvar name=2>{{MW lts release number}}</tvar> ([[LTS]]).

Sound reasonable?

It does to me. The way it is now is a bit confusing. I guess it's less inaccurate than previously but it makes one (me) wonder why the same versions are mentioned twice.

Altered.

new link is https://gerrit.wikimedia.org/r/admin/repos/q/filter:mediawiki%252Fskins

 Done

I couldn't find anything related here.

I wanted to use git to download the 1.35.1 tag but with --depth=1. I think it is workth mentioning on Download from Git#Download a stable branch that (from what I found on google) this is the command:

git clone https://gerrit.wikimedia.org/r/mediawiki/core.git --branch 1.35.1 --depth=1 mediawiki

so instead of cloning a actual branch, we can use the tag.

How did I get to this you may ask? Well when first doing:

git clone https://gerrit.wikimedia.org/r/mediawiki/core.git --branch REL1_35 mediawiki --depth=1

and than

git checkout 1.35.1

(as the page suggests) you wont get anywhere, because no tags where downloaded.

Am I missing something? Let me know please.

or is it so that --branch REL1_35 already is on tag 1.35.1? But than why should I checkout 1.35.1?

Try the following:

git clone -b 1.35.1 --single-branch  https://gerrit.wikimedia.org/r/mediawiki/core.git mediawiki

Never mind, it looks like using git to specify a single tag as the branch is the quickest and most light-weight checkout.

Checking out the entire repo and then fetching just that tag takes me about half a minute longer and 40mb more space than just checking out a single branch:

 time (
   git clone https://gerrit.wikimedia.org/r/mediawiki/core.git &&
   cd core &&
   git checkout 1.35.1
 ) && du -sh core
 ...
 user    5m11.114s
 ...
 448M    core

vs.

 time git clone -b 1.35.1 --single-branch \
   https://gerrit.wikimedia.org/r/mediawiki/core.git &&
 du -sh core
 ...
 user    4m40.874s
 ...
 407M    core

But your method is fast and cheap.

 time git clone -b 1.35.1 --depth 1 \
   https://gerrit.wikimedia.org/r/mediawiki/core.git &&
 du -sh core
 ...
 user    0m2.076s
 ...
 174M    core

Thanks, but I just googled ;)

Did you maybe see my second question? I’m a bit confused, if just clone the branch, am I already on 1.35.1? If so than I could also clone the branch with depth=1.

A branch is not the same as a tag. Commits can be on a branch after the revision tagged 1.35.1. If you want 1.35.1, it is best to directly check it out.

so as I thought, on both (-b 1.35.1 and -b REL1_35) I am greeted with the initial setup for 1.35.1.

Sure, as you said, if you really want the exact version from the release-date, use the tag. But I guess when I stay on the branch REL1_35 and there comes version 1.35.2 around, I will probably only have to run git pull --recurse-submodules to get the latest code.

but that is my problem with the page: it suggest using --depth=1 (with -b REL1_35), but when doing so, I can not later git tag -l | sort -V and git checkout 1.35.1.

The text needs to be written like that: If you want clone a tag (1.35.1) AND use --depth=1 you have to clone it directly with -b 1.35.1.

Or am I totally in the wrong here?

Last night I had to pull get branches for a shallow clone, so I ended up looking for the answer.

The answer is that you have to tell git where to get branches from:

 git remote set-branches origin '*'
 git fetch

After that, you'll be able to check out branches from a repo that you originally created with a shallow clone.

well at first I think we are talking about two different problems:

1. Problem: is it better to use branches (REL1_35) or tags (1.35.1). I think we should at first clarify on the article that a: there are two ways to clone the stable branch/tag and b: it is recommended to always clone the current stable tag instead of the branch. Would you agree? So when 1.36 comes along, it would be better to clone --branch 1.36 instead of REL1_36 (i still find it confusing that --branch is used for branches and tags qually. that means you can't have a tag with the same name as a branch, right?)
2. Problem: use of depth It is not clear from the description of the page that you can't use git tag -l | sort -V after you cloned with depth=1! So I would explain it in a way that you either clone without depth and than be able to switch tags or use depth=1. of course (thanks for that) here is your command coming in handy (when cloned with depth=1) git remote set-branches origin '*' git fetch

Did I miss something? :)

i still find it confusing that --branch is used for branches and tags qually. that means you can't have a tag with the same name as a branch, right?

I hadn't thought about it, but I think you're right.

Did I miss something?

I think you've got it.

So I tried to make update. git pull in the mediawiki-folder worked fine. but than the page says:

The new version of core may require newer versions of extensions and skins, so you must go into each extension and skin directory and update it with a command like git pull --recurse-submodules.

so I went to extensions/ and into any folder and run git pull. I always got this error:

You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.
git pull <remote> <branch>

Did I miss something?

Switch to a branch before using "git pull".

Thanks. But I wonder if this is necessary? I already pulled the core-repo. Shouldn't there be a command like git submodule update --init to also update all the sub-modules? I don't think it makes sense to go into every extension-subfolder/module and switch branch and pull manually.

Maybe git submodule update --recursive --remote ?

When you checkout a submodule (for example, Vector via git submodule update --init skins/Vector), the repository for the submodule is on a specific commit, not a branch. That is the cause of the You are not currently on a branch message.

Release branches for MediaWiki have submodules for all the extensions but the measter branch does not. Which means that, yes, you can use git submodule update --init --recursive on release branches for those extensions included in the release branch, but not on master, as a general rule. (That last submodule command will not change anything if your parent repository is on the same revision as it was when you initially checked out the submodules.)

The exception to this rule for master is if you used git to create submodules for the extensions you install. If you add the Foo extension via git submodule add -f http://example.com/foo.git extensions/Foo, then you can update it later using a git submodule command. If you just want to update the already checked-out branches for any submodules, then the command you're looking for is git submodule foreach git pull.

ok, I think I should have clarified that I not intend to be on master. sorry.

I want to clone stable:

1. clone -b 1.35.1 --depth=1
2. git submodule update --init

That means I have also the bundled extensions with stable version.

My question now is: Should I update the bundled extensions from time to time when they release new version (or don't they?)?

Or should I wait for the next stable release of mediawiki where I will clone the new bundled extensions?

ok, I think I should have clarified that I not intend to be on master. sorry.

No, I understood that. What I said stands.

Should I update the bundled extensions from time to time when they release new version (or don't they?)?

If any of the bundled extensions has a back-ported fix, the MediaWiki version will change. Because of the development that most of core MW and the bundled extensions have used, they tend not to back-port anything but security fixes. Announcements for these updates are made on the announcements mailing list.

So, unless you want to spend the time to follow master or the WMF branches, you don't have to worry about upgrading your extensions outside of the MW update cycle.

I have a question regarding /vendor (#Fetch_external_libraries)

I am downloading with

• git clone https://gerrit.wikimedia.org/r/mediawiki/core.git --depth=1 --branch 1.35.1 mediawiki

• git submodule update --init --recursive

When I look under /vendor, I can see that it is not empty. Is it possible that in my case it is not necessary to use composer or download from git once more?

From what I understand I only need composer for additional extensions?

Right, MW's git repo has a vendor submodule. If you just use that git checkout, you don't need to run composer update (unless you want to update one of the dependencies).

I suggest there be an addendum created on this page (and others) indicating when a standard has changed -- links, for example. Doing so would allow users to find this page through a search engine when querying the older style links that are used in a plethora of non-updated examples. This would save others from the repeat of my unproductive experience:

I was following this page: https://stackoverflow.com/questions/49267591/how-to-best-add-extensions-when-using-official-docker-image-for-mediawiki in learning how to add some extensions but got tripped up on the old standard of linking to the code repository:

gerrit.wikimedia.org/r/p/mediawiki/extensions

whereas it is now:

gerrit.wikimedia.org/r/mediawiki/extensions

Most older examples accessing the repository incorporate the old standard link which no longer work and do not lead here.

We can't really fix other people's outdated sites, sadly. There was quite a lot of noise made at the time of the gerrit upgrade two(?) years ago that /r/p/ links won't work any more. The more banners and warnings you put on these pages, the more readers will ignore them entirely, or miss the important bits.

I'm just trying to downloading all extensions, following this sequence:

git clone https://gerrit.wikimedia.org/r/p/mediawiki/extensions.git
cd extensions
git submodule update --init --recursive

The problem is that some extensions asks for authentication. I've made account in https://gerrit.wikimedia.org but it is 'Unauthorized' and result is:

$ git submodule update --init --recursive
Cloning into 'Notifications'...
Username for 'https://gerrit.wikimedia.org': Spas.Z.Spasov
Password for 'https://Spas.Z.Spasov@gerrit.wikimedia.org':
remote: Unauthorized
fatal: Authentication failed for 'https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Notifications.git/'
fatal: clone of 'https://gerrit.wikimedia.org/r/p/mediawiki/extensions/Notifications.git' into submodule path 'Notifications' failed

(I found that extension '2ColConflict' also make this trouble...)

Could someone help me get through this?

Same problem to me. Notifications asks username and password.

@Spas.Z.Spasov I did

cd extensions/
git submodule foreach 'git checkout -b REL1_28 origin/REL1_28 || :'
git submodule update --init --recursive

And now it works.

Thank you! I will try it these days. And will report also.

Are those "translations" of commands really work in Git softwares?