It would be great to provide a mechanism for disabling extensions on the fly - as far as I know, Composer doesn't provide a way to un-include an extension without actually uninstalling it. The only solution I can think of is to have a global variable called something like $wgDisabledExtensions, settable within LocalSettings.php, that would hold an array of extension names. Any extension included using Composer would be expected to check that array right in the beginning, and, if its name was included in the array, to exit out immediately. What do people think?
Topic on Talk:Requests for comment/Extension management with Composer
+2 I also strongly believe that it must be possible to prevent the autoloading of extensions installed via Composer. I do not know what the best implementation of this will be, but Yaron's idea sounds good to me.
As I have already indicated on the SMW mailing list, I think this approach is conceptually incompatible with dependency management. Not loading something that is installed undermines the safety of the system. It's like installing packages via apt-get and then disabling it manually somehow in such a fashion apt-get does not know it is no longer installed.
I do not want to repeat directly what was written on the mailing lists, but they provide an idea of the issues and concerns raised:
[1] thread of the smw developer mailing-list
[2] thread of the smw user mailing-list (starting with egel at Jan 18, 2014; 10:10pm)
Jeroen - could you clarify in what way safety would be undermined, or give an example?
Install A, disable it. Install B that depends on A. No problem is detected since A is "installed". Wiki breaks because B misses its dependency.
Ah, I see. Well, I imagined that extensions would also check for their dependencies - so that an extension that required Semantic MediaWiki (the obvious example) would also exit gracefully if SMW were included in the "disabled" array.
Dependencies are transative. Do you want SMW to check for all its dependencies? That breaks encapsulation, and gets really ugly as soon as you use components you do not maintain yourself.
Also, in case someone then runs into an error because their installation is broken (rather than a tool catching the issue before breaking the install), what are they going to do about it? If you have the disabled thing, I see some users trying to do a manual install. Or perhaps install a wrong version.
If you break all that security, you can forget about relyable automation. So bye bye idea of having a GUI extension installer on top of the existing Composer support.
Well, I probably should have been more detailed with my proposal. I'd imagine that only "user-facing" (if that's the right term) extensions would be disable-able; so if someone added, say, Validator/ParamProcessor to the the array of disabled extensions (one of SMW's dependencies), that would simply be ignored. I don't think it would be that hard to have extensions handle this variable in a smart way.
I also don't see the connection between this and security and automation; although maybe that's no longer relevant, given the above.
> I don't think it would be that hard to have extensions handle this variable in a smart way.
What you are saying is that $wgDisabledExtensions becomes intelligent or the extensions that handles this variable should attach some logic in order to handle dependency knowledge?
While I could agree to have $wgDisabledExtensions as debugging tool to temporary disable a extension, I surely can't see $wgDisabledExtensions as tool that would circumvent the intention of Composer namely the management of dependencies.
Removing a package from the composer.json and therefore from the deployment tree is the safest method because it ensures that dependencies are properly resolved and not just disabled.
I guess I'm saying the latter - I just mean that a simple logic needs to be in place: if the current extension is listed in the array, exit; if a "real" (non-library) extension that this extension requires is in the array, exit; otherwise, don't exit. What's unsafe about this approach?
Yaron: can you make a clear case for why it is problematic to manage what is installed purely via Composer? I simply fail to see the real problem there. And without that any undermining of the current system is not going to seem worth it.
Sure, I can think of two obvious examples: running a wiki farm, and debugging one's own installation. In both cases, the admin might want to disable extensions in any combination - well, almost any, barring of course extensions that depend on another. As far as I know, there's no way to do that currently with Composer; so for the debugging, one would have to keep re-installing different combinations, while for the wiki farm you'd have to literally have a different set of code for each possible combination - annoying if you have more than one extension, untenable if you have more than, say, three. That's not to mention the difficulty it would place on maintaining any local changes to the code.
Also, can you explain in what way my proposed system would undermine things?
> running a wiki farm, and debugging one's own installation.
I do not see the actual problem in either case. You are guessing some stuff will be very hard or not possible. However if you know how to use this tool I think you can do everything you need to do without to much bother. Some things will get a bit more cumbersome, though at the same time others become less so.
> so for the debugging, one would have to keep re-installing different combinations
In some cases you might want to do this yes. Though is removing the thing from the composer file and running composer update really that hard? It's slightly more work than commenting out an extension. Though at the same time it will also stop loading not needed dependencies automatically, which you'd have to do manually otherwise.
Also, disabling extensions is one thing you might want to do during debugging (though not something you do every minute I'd hope). Another thing is switching between different versions. That one is actually easier with Composer.
> wiki farm you'd have to literally have a different set of code for each possible combination
Yeah so? Disk space is a non-issue. I personally would prefer this approach if I ran a wiki farm, it's way less scary than running different wikis with different config and even different extensions via "the same" code.
> That's not to mention the difficulty it would place on maintaining any local changes to the code.
If you do not manage your changes via git, then yeah, things could get difficult. (Though in many cases you could still do what you want to do without to much hassle). The only place where this gets really annoying is if you indeed have a farm and need the changes at all places. In which case you can just stuff them in git and have much more solid control over your farm than you'd otherwise have as well.
> Also, can you explain in what way my proposed system would undermine things?
I have already done so in a previous post.
Come on, you can't seriously suggest that someone running a wiki farm set up 32, 64 etc. separate installations, one for each combination of extensions you might want to disable, is a workable solution? What if the code includes some custom changes? It's a nightmare just to think about it.
Similarly, if someone is uninstalling extensions to do debugging, and any of their extensions contain custom changes, that becomes quite a bit of work as well - and I would think they easily run the risk of deleting important work.
I know you explained the undermining thing before, but I'm pretty sure that was based on an incorrect understanding of my proposal, as I noted. Could you try explaining it again?
I think you both have very valid points.
Having every extension check a config variable and disable itself is not a solution in my opinion. Some will do it, some won't. It will get messy. And it will get really messy once you get to dependency trees that are more than one or two levels deep. The whole point of dependency management is to not have to check if everything is in place. Take the Lingo extension. It depends on ext-dom (not a very good example, i know). Now suppose somebody switched that off. Lingo would consequently not load. However, SemanticGlossary would, because all its dependencies are there. And yes, you could put all sorts of logic into maintaining that config var of switched off extensions, but doing that in each and every extension is just wrong.
Would it be possible (both technically and "politically") to change composer, so that you could load and install a library, but switch it off, so that the library itself and all depending libraries are not included in the autoloader? This way you would only need one set of libraries for all wikis of a farm and only the autoloading scripts need to be different.
F.trott - yes, that's not a great example: I'm not even sure what ext-dom is, but I'm assuming it's some sort of library, not meant to be used standalone, so you couldn't even disable it under my proposed scheme. However, you do maintain another extension, Semantic Forms Inputs, that does fit the bill. And for that sort of extension (and maybe for Semantic Glossary as well, if I'm wrong about it), I would say (a) it's not a huge deal to have extensions check for all their dependencies, if they have more than one, and (b) even if they don't check for all dependencies, it's not a big problem: if the admin disables an extension but forgets to disable an extension that depends on it, they'll get an error message - which is the same thing that happens right now if an admin comments out one extension but forgets to comment out the extensions that depend on it. So the situation won't be any worse.
If there were a true Composer-based solution, that would be ideal, of course.
> (a) it's not a huge deal to have extensions check for all their dependencies
The extension shouldn't have to check dependencies because that's the job of the Composer otherwise you have two components that do similar things, the Composer and the extension.
> it's not a big problem: if the admin disables an extension but forgets to disable an extension that depends on it, they'll get an error message
You arbitrarily disable a component (or package) knowingly at the cost of incompatibilities or even instability. That's a questionable approach!
Yaron, I don't say that your scheme will not work. It may, technically, in an ideal world. It requires that each and every extension developer plays along nicely and (1) checks all their dependencies and (2) turns their own module¹ off if it's in the config var. Fat chance, really.
I mean, as a developer why would I check for dependencies, if I know that's taken care of by composer. After all I carefully crafted that blasted uncommentable² json. And why would I turn my extension off for you? What's my benefit? Heh, some developers will not even know that that was required. And they won't notice either, all the intended functionality would be there.
No, IMHO you need a solution that is enforcible, i.e. something that does not require extensions to play along. If you wanted you could probably write some central manager that needs to be called by extensions for them to be enabled. And that central manager could then decide to not enable an extension based on some criterion.
I'd rather try getting a patch into composer than trying to get some extension manager into MediaWiki.
We can not be the only ones with this issue, can we? How do other frameworks handle this?
¹ Yes, module. I don't buy the "only visible extensions" part. It's an arbitrary definition. ² You actually can misuse undefined key-value-pairs to put comments in the composer.json.
MWJames, F.trott - I disagree with some of your specific objections, but I agree with both of you that my proposed solution has weaknesses. However, it's the only solution I could think of - if there's a better one, whether it's directly part of Composer or not, I'd be glad to switch to that one instead.
Come on, you can't seriously suggest that someone running a wiki farm set up 32, 64 etc. separate installations, one for each combination of extensions you might want to disable, is a workable solution? What if the code includes some custom changes? It's a nightmare just to think about it.
Each wiki would have their own composer.json file. And if you have wikis that have the same config, they could share the file. All you then need to do is keep the composer.json files up to date (which is likely easier than managing this in one huge file with switches in it). And run "composer update" in each wiki dir. That does not sound very hard to me. That includes handling custom changes you made.
So yes, I seriously suggest this. And I suggest you learn how this tool works and can be used, before you conclude it cannot fit in a particular workflow.
If you ran a wiki farm with 100 wikis in it, and you wanted to make a local change to some Composer-based extension, would you need to make the change once, or 100 times? (Or somewhere in between, if you tried to group together extensions that had the same configuration of extensions?)
You'd need to make it once.
Can you document this setup you're talking about somewhere, or at least explain it in more detail? I don't understand how you can have multiple composer.json files modifying the same set of code. (If I understand even that part correctly.)
Indeed, an explanation would be nice. I do understand the management of dependencies of composer, and I can see that wiki farm configurations must be handled in a new manner with composer. But I cannot see how one could split the extensions of a farm into two sets: commonly used extensions and individually used extensions added to the commonly used extensions. Any idea for solving the following problem?
- wiki farm with 100 wikis
- all wikis share a basic set of extensions
- some wikis add further extensions which are not activated on the whole farm (example: BreadCrumbs)
Solution with LocalSettings.php
- one directory with all extensions (including BreadCrumbs)
- one file
globalExtensions.php(containing therequire_once-statements for the common set of extensions) - individual files
localExtensions.php(containing therequire_once-statements for individually chosen extensions) - individual
localSettings.phpfiles (including the filesglobalExtensions.phpandlocalExtensions.php) - adding/removing of a globally used extension:
- adding/deletion of
require_once-statement inglobalExtensions.php - adding/deletion of extension from the global extensions directory
- adding/deletion of
- adding/removing of a locally used extension:
- adding/deletion of
require_once-statement inlocalExtensions.php - adding/deletion of extension from the global extensions directory
- adding/deletion of
?? Solution with Composer (assuming that in the long run all extensions will use Composer)
- one directory with all extensions (including BreadCrumbs)
- one file
composer.json(not including BreadCrumbs) - link of individual wikis to the global
composer.jsonfile - ?? BUT my question is: how can individual wikis ADD BreadCrumbs to this scenario?
- adding/removing of a globally used extension: ok - treatment via composer
- adding/removing of a locally used extension ???
I am aware of the fact, that one could argue, that there might be dependencies between globally and locally used extensions - but adhering strictly to this credit would mean that wikis on a farm cannot be indiviualized, unless all possible combinations of individualizations are spelled out in separate composer.json files and each wiki links to a different composer.json file - which makes the idea of maintaining a wiki farm rather odd
PS I have chosen BreadCrumbs as an example of an extension which changes the appeareance of a wiki if it is activated
Does anybody have a solution for this scenario? Does composer have an inheritance mechanism?
This post was posted by Kappa~mediawikiwiki, but signed as Kappa.
Useful summary, despite the blanks about composer: can you integrate it into Manual:Wiki family? Thanks.
I can come with a POC for wiki farms – I tested it with my wikifarm extension – but it’s very hacky. For the context, let’s say you have a single directory with MediaWiki code ($IP) and two separate directories, one for each wiki ($D1 and $D2), and you want to install SemanticMediaWiki with Composer only for the first wiki (but you have to execute Composer for both wikis given MW core has Composer dependencies).
$ cd $IP $ vi composer.json (and add in config: "vendor-dir": "$D2/vendor") $ composer update --no-dev $ vi composer.json (and change in config: "vendor-dir": "$D1/vendor") $ vi composer.json (and add in require: "mediawiki/semantic-media-wiki": "*") $ composer update --no-dev $ mkdir vendor $ vi vendor/autoload.php (and add your logic of selecting the wiki followed by require_once "$D1/vendor/autoload.php" or "$D2/vendor/autoload.php")
At the end, you have the real Composer data in $D1/vendor and $D2/vendor. When executing MediaWiki, the only visible difference (that I see) compared to a classical installation is MediaWiki doesn’t show Composer libraries in Special:Version.
This type of hack fullfills the issue with dependencies said by Jeroen since each Composer execution is kept and hence is a coherent set, and it solves the basic need for a wiki farm. Obviously it stays hacky and it needs some work and tests to integrate it in a real wiki farm. An issue is you have to execute the “biggest set” (here MW+SMW) in last, else SMW will be removed from extensions/ (I know, “biggest” is badly defined since the order is not total).
EDIT: in fact, I now see Composer librairies in Special:Version. Don’t know what happened.
@Nemo bis @Yaron Koren @Jeroen De Dauw @F.trott @MWJames @Kghbln @MarkAHershberger @Mglaser @Cindy.cicalese @Egel @Nicolas NALLET: I revive this thread and ping you since I just did, I dare to think, a breakthrough on this issue, after thinking and working during some months about it.
I wrote a summary of the issue (from this thread and personal ideas and experiments) on Requests for comment/Extension management with Composer/Farms and Composer; and I just finished a proper implementation of what I think is the better compromise to manage extensions with Composer in a farm (4th method in the summary page). This allows to activate and deactivate the Composer-installed extensions on a per-wiki basis in a farm, and there is almost no overhead, though it has to be checked in a production environment. I’m interested in any comment you have about the summary/methods and/or the MediaWikiFarm extension.
Although this method works, this stays fragile; and I personally greatly prefer non-Composer-installed extensions, or at least when the activation is independent from Composer, like did PageForms 4.0 and SemanticFormsSelect 2.0 (for the some extensions I’m aware).
It would also be helpful to get input from @Legoktm
This talk is mentioned at Requests for comment/Improving extension management