Talk:Requests for comment/MediaWiki libraries

Composer managed contents

In my implementation for Requests for comment/Structured logging in Gerrit change 119939 I created a libs directory that is managed using Composer. Would getting that implementation merged be a reasonable resolution to this RFC? The concept of this form of library manage was vetted and approved by Tim in his review of that RFC. --BDavis (WMF) (talk) 22:12, 23 April 2014 (UTC)

Yep, that solves it.--Ryan lane (talk) 22:24, 23 April 2014 (UTC)

Discussion on April 29

We'll be discussing this RfC Tuesday 29 April, 2200-2300 UTC. Sharihareswara (WMF) (talk) 17:40, 25 April 2014 (UTC)

Summary of discussion: We just need to merge "Add Composer managed libraries" https://gerrit.wikimedia.org/r/#/c/119939/ in order to complete part I of this. Then Part II is actually moving everything else into that directory, and also making it reasonable to move some extensions into that directory. Then we started talking about the Composer RfC. :-) Log:

22:02:57 <sumanah> OK, now on to the first topic today, Ryan_Lane's MediaWiki libraries RfC. Then we'll move on to parent5446's RfC on 3rd-party components

22:03:01 <sumanah> #topic MediaWiki libraries

22:03:12 <sumanah> #link https://www.mediawiki.org/wiki/Requests_for_comment/MediaWiki_libraries Ryan Lane's proposal (better handling and versioning libraries that are MW extensions)

22:03:12 * marktraceur frows at Book Management

22:03:36 <sumanah> #chair sumanah brion andrewbogott TimStarling

22:03:36 <wm-labs-meetbot> Current chairs: TimStarling andrewbogott brion sumanah

22:03:41 <parent5446> So I have a question about this RfC. It mentions LdapAuthentication as an example. What library in LdapAuthentication would we be packaging with core?

22:03:46 <sumanah> #info See https://www.mediawiki.org/wiki/Talk:Requests_for_comment/MediaWiki_libraries - we just need to merge "Add Composer managed libraries" https://gerrit.wikimedia.org/r/#/c/119939/ in order to complete part I of this.

22:03:47 <sumanah> #info Then Part II is actually moving everything else into that directory, and also making it reasonable to move some extensions into that directory.

22:04:04 <parent5446> B/c I understand the RFC in the context of the Gerrit patch

22:04:19 <parent5446> Libraries that are being used in core will need to be managed somehow

22:04:23 <Ryan_Lane> parent5446: a number of extension use LDAP as a library

22:04:31 <brion> i’m not sure i understand this rfc; is the idea to package specific libraries in mediawiki core?

22:04:39 <Ryan_Lane> I'd split the necessary functionality out of the LDAP extension into a common library

22:04:44 <brion> or to provide a general place to put libraries?

22:04:47 <parent5446> ...how? Are there extensions that use LDAP other than LdapAuthentication?

22:04:53 <TimStarling> I think the idea is to create a separate class of extensions that you put in libs/ instead of extensions/

22:05:05 <Ryan_Lane> TimStarling: yep

22:05:13 <brion> first question: why?

22:05:25 <brion> second question: what needs to be different other than the filename ‘extensions’ -> ‘libs’?

22:05:36 <Ryan_Lane> because there's now a lot of extensions being used as libraries and they aren't being versioned with mediawiki

22:05:40 <parent5446> OK, but the gerrit patch makes it look like we'd be packaging them with core. So it makes it sound like we'd be packaging libraries within core that are not actually used in core.

22:05:45 <brion> third question: does that patch adding a libs directory help or hinder this rfc’s recommendations?

22:05:53 <sumanah> (the patchset is by bd808 in case he wants to comment)

22:05:57 <Ryan_Lane> most of these could be bundled with core

22:06:11 <TimStarling> we have includes/libs already for things that are bundled

22:06:13 <Ryan_Lane> and should definitely be versioned with releases

22:06:28 <Ryan_Lane> TimStarling: that's for things that core depends on, but not for things extensions may need to depend on

22:06:45 <brion> so is this directory’s contents something is fixed, that extensions can depend on always being present?

22:06:55 <brion> or is it something extensible, that extensions can rely on having a way to install things there?

22:07:09 <parent5446> I don't really think it's a good idea to package extension libraries as part of core if core is not using it.

22:07:23 <bd808> The best solution for Ryan's RFC would probably be enabling the full use of Composer (or a similar system) with wm-core

22:07:27 <parent5446> Maybe an extension needs a different library version, and can't use it because core decided to package it

22:07:48 * sumanah invited Jeroen to discuss composer-y stuff with us in this chat but - ah, there's JeroenDeDauw!

22:08:03 <JeroenDeDauw> Just saw calendar notification now

22:08:07 <JeroenDeDauw> Anout to head off

22:08:14 <Ryan_Lane> so, maybe I should describe my issue, rather than this specific solution?

22:08:18 <sumanah> JeroenDeDauw: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/20140429.txt starting at 2200 in case you wanna see our talk till now

22:08:26 <sumanah> s/talk/chatter/

22:08:29 <TimStarling> with composer, I'd like to know whether you think extensions should be in mediawiki/* or mediawiki/extensions/*

22:08:32 <brion> Ryan_Lane: yes please :D

22:08:42 <TimStarling> in the package names

22:08:44 <sumanah> #info the Composer RfC: https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_with_Composer

22:08:57 <Ryan_Lane> so, we have extensions. those extensions for a long time only had a dependency on a specific mediawiki version

22:09:02 <TimStarling> because JeroenDeDauw put a few in mediawiki/*, but it seems a bit weird to me

22:09:14 <Ryan_Lane> now we have extensions that only exist as libraries for use by extensions

22:09:14 <JeroenDeDauw> TimStarling: vendor/package

22:09:18 <JeroenDeDauw> TimStarling: two levels

22:09:24 <brion> (many extensions require external dependencies as well…)

22:09:52 <mwalker> extensions can also be hooks onto other extensions -- but that doesn't make the 'parent' extension a library

22:11:03 * aude waves

22:11:09 <TimStarling> hmm

22:11:12 <mwalker> this seems to be mostly an argument about our release process... in theory, everything in a release is stably versioned and has release branches for backports

22:11:43 <Ryan_Lane> well, it's more an issue that we have no reasonable way to specify extension compatibility

22:12:00 <Ryan_Lane> and now we need to specify extension compatibility with mediawiki and with other library extensions

22:12:02 <parent5446> I mean, this is basically what composer was made to do

22:12:21 <brion> Ryan_Lane: don’t you just grab them from the same version branch?

22:12:21 <parent5446> Except people don't want to use composer in core

22:12:34 * sumanah sees parent5446's point. If this is a configuration/dependency management problem, maybe we should go with the thing that does that

22:12:40 <Ryan_Lane> brion: very few people use version branches

22:12:52 <brion> Ryan_Lane: how would this solve that?

22:12:59 <mwalker> we're cutting the version branches automatically now

22:13:14 <greg-g> mwalker: but everyone else still uses tarballs

22:13:15 <Ryan_Lane> my proposal was to have the common libraries shipped with mediawiki and locked with the version

22:13:22 <brion> …

22:13:31 <brion> so something that’s common to a group of extensions, and has no relation to mediawiki core...

22:13:34 <brion> … would be shipped with core?

22:13:38 <parent5446> Shipping common libraries is kind of like a band-aid to the actual problem

22:13:39 <TimStarling> I think it's fine to have extensions that depend on other extensions

22:13:50 <bd808> parent5446: I think we are open to using composer in core, but new need to figure out how to do that in a way that works for JeroenDeDauw's use case and mine (which is basically the same as Ryan_Lane's)

22:14:07 <Ryan_Lane> the issue is that most of these library extensions probably should just be core

22:14:09 <TimStarling> drupal has a DIY system for this

22:14:18 <TimStarling> or of course we could use composer

22:14:25 <parent5446> I thought the main problem was that people didn't like how it didn't interact properly with Git?

22:14:29 <Ryan_Lane> if 20 extensions are using an extension, why isn't it a core feature?

22:14:30 <sumanah> Emufarmers: Dantman: would love to have your input here as people who work on non-WMF MediaWiki installations

22:14:32 <TimStarling> or support any other package management system, like PEAR or .deb or whatever

22:14:50 <brion> Ryan_Lane: are those 20 extensions all specific and only used by a tiny minority of installations?

22:14:54 <mwalker> greg-g, we bundle extensions into our release tarballs...

22:14:55 <brion> or are they used all the time by most?

22:15:05 <Ryan_Lane> to be fair, I'd be fine with composer, if it can be used reasonably

22:15:23 <parent5446> Well many extensions have already been made to work as composer libraries

22:15:24 <TimStarling> if a facility is widely used, I think it can be in core, you know that I am not one of the people who has a narrow view of what core should be

22:15:25 <greg-g> mwalker: yeah. (/me doesn't want to derail, it was a minor point)

22:15:27 <brion> it feels like the solution here is “a dependency-managing installation system for extensions"

22:15:38 <Ryan_Lane> TimStarling: I'm not saying you are :)

22:15:42 <TimStarling> but yes, we need extension dependencies for various reasons

22:15:42 <sumanah> Which means we should check out https://www.mediawiki.org/wiki/Requests_for_comment/Extension_management_with_Composer

22:15:51 <sumanah> and maybe expand it

22:16:04 <Dantman> sumanah: I'm not 100% certain of the topic, or even whether "extensions as composer modules" is included in this or excluded.

22:16:18 <TimStarling> not just for libraries -- some extensions hook others, for example

22:16:28 <Ryan_Lane> yep

22:16:41 <Ryan_Lane> it's something we've been punting on forever

22:16:43 <sumanah> Dantman: sounds like we're talking about including it, maybe

22:16:44 <TimStarling> some use RL modules from another extension

22:17:09 <sumanah> Dantman: the topic is https://www.mediawiki.org/wiki/Requests_for_comment/MediaWiki_libraries Ryan Lane's proposal (better handling and versioning libraries that are, or are used by, MW extensions)

22:17:22 <Ryan_Lane> so, let's drop this proposal and maybe talk about the composer suggestions?

22:17:43 <Ryan_Lane> or is that not how this process works? heh

22:17:52 <parent5446> Well if we need to itemize this to dependencies on individual RL modules and hooks, there's not really any system that handles that

22:18:05 <parent5446> With composer it's either you depend on the entire extension or you don't

22:18:12 <Dantman> Well this idea of extensions as composer modules has annoyed me over and over again.

22:18:14 <parent5446> Unless you separate into further libraries

22:18:15 <sumanah> Ryan_Lane: I'm fine with the process working like that, if you want to abandon your RfC and become a coauthor on the Composer one :)

22:18:53 <bd808> Extension management with composer would allow managing dependent libraries as well, for the extensions. My use-case of wanting an external library for core is slightly different, but could pervert the system by using a basically empty extension just to get the library management.

22:19:00 <sumanah> andrewbogott: I guess one question for you as Ops representative is how Ops feels about Composer (although I assume if Ryan's fine with it then you would be too)

22:19:02 <Ryan_Lane> wasn't that another RfC that was in the queue for today?

22:19:21 <sumanah> Ryan_Lane: no, it wasn't; it is one that I wanted people to have fresh in their memory because I predicted it would interrelate

22:19:24 <Ryan_Lane> the current implementation of composer doesn't fit with our deployment process

(And then we moved onto the Composer RfC proper.) Sharihareswara (WMF) (talk) 23:42, 29 April 2014 (UTC)

Dependency management

It seems like the root problem is the lack of dependency management in extension installation when manually fetching extensions instead of grabbing everything from a consistent git checkout.

I don't think stashing random libraries unused by core into MediaWiki would really help here, while some tool for managed installation and updates would be more useful. --brion (talk) 22:17, 29 April 2014 (UTC)