Talk:Requests for comment/pywikibot 2.0 packaging

no library i18n

I think it is easiest if we remove the use of i18n from pagegenerators for 2.0, and design i18n into the next major release of the library. John Vandenberg (talk) 09:10, 29 November 2014 (UTC)

i18n for pagegenerators was an initial sample for multilingual development according to phab:T57109. There are additional config settings like userinterface_lang to get the right language for bot operators.  @xqt 12:47, 30 November 2014 (UTC)

I agree with xqt, i18n is a vital part of pywikibot, I think separating i18n to two parts of "vital i18n" (any pywikibot related i18n + terminal i18n) and "optional i18n" (or scripts i18n) is a better idea Ladsgroup (talk) 11:47, 1 December 2014 (UTC)

two packages

I think the simplest approach for 2.0 is to maintain two packages: 'pywikibot' (the library) and 'wikimedia-pywikibot' which includes the scripts (as pywikibot.scripts) and regularly updated family classes. Labs and wikimedia home-bot operators would install 'wikimedia-pywikibot', but non-WMF users would install 'pywikibot'.

Then we can plan to remove the worst of the family file issues by 2.1, meaning we only have the 'scripts' problem to worry about, and we have some time to resolve that properly. John Vandenberg (talk) 09:10, 29 November 2014 (UTC)

These parts of bot classes in script folder which are included by other scripts should become part of a library in pywikibot folder. E.g. add_text.py has identical parts of cosmetic_changes.py and the last one has a lot of wikitext manipulating methods which should be library parts. I am unsure about the two packages. A lot of the scripts can be used outside Wikimedia sites and I don't see any reason to split them. Maybe there are good reason to make them L10N undependent from a given site. This is also valid for Wikimedia projects. A lot of scripts are designed for Wikipedia sites and it is on bot owners side to localize them.  @xqt 12:58, 30 November 2014 (UTC)

separately packaged wikimedia family files

I've been playing with packaging the wikimedia family files separately, and it is feasible. My first attempt requires intrusive changes to the bootstrap / config code, as the list of valid families is needed to validate the families used in the config (e.g. config.usernames), and a separate package needs to be able to 'import pywikibot' (which in turn loads config), causing a cyclic dependency between library and families package. But it is doable, and could be made much less intrusive. John Vandenberg (talk) 09:10, 29 November 2014 (UTC)

That would be my suggestion too. So everytime we need to update the family files we could just publish a new version of that. Also this would give us a chance to phase that out (slowly of course) and as soon as we don't need family files anymore (at least no default) users could just use the “main package”. Separating it could be a problem of course. — xZise [talk] 11:05, 29 November 2014 (UTC)

If we're going to have separate packages, we need a simple way for users to update.

The 'easy' solution is to have a 'wikimedia-pywikibot' package, which only exists to depend/install the other required packages? But that means releasing two packages any time one package needs to be updated. More release work, though it could be automated, it is still more chance of mistakes being made. John Vandenberg (talk) 11:54, 29 November 2014 (UTC)

But the family files are not that often changed and just because pywikibot is updated, we don't also need to update the pywikibot-families or whatever this is going to be, especially if there were no changes made to the family itself. Isn't the question how we are going to get family related changes as fast as possible to the users, and this would be a solution.

The other of course would be to have that in the pywikibot package itself as I don't think that we need to fix those files that often. If you look in the families folder all files are almost at least a month old. And the more recent changes were not specific to the family file but are related to changing pywikibot itself, so we would have to release a new pywikibot version anyway. Changes like https://github.com/wikimedia/pywikibot-core/commit/ff69b61805ada9a60995fe3d061c033822f71962 could be delayed until a next release of pywikibot.

Also we don't want to break other people's scripts so we usually don't break anything so if we apply changes to pywikibot which would affect the families, those should work anyway as we won't break it. But I'm no expert about how packaging works so maybe it's not that easy to advance the pywikibot version without updating the repositories of pywikibot-families. — xZise [talk] 17:31, 29 November 2014 (UTC)

The family files arnt being updated regularly because I asked for the bot updating them to be stopped as it is solving the wrong problem, the wrong way. See scripts/maintenance/wikimedia_sites.py , and the log of wikipedia_family.py for 'language.*size' changes - they are usually a few each month, and without them interwiki.py breaks badly, and wiktionary bot operators scream (most other WMF projects use wikidata instead of interwikis) . We've nearly solved that problem with dynamic interwikis, I need to refresh a patch for dynamic Family.languages_by_size, but we also need dynamic Family.langs (several patches are up for review, but those dont completely solve the problem) and dynamic Family.obsolete. However, wiktionary will probably become a Wikidata client in 2015, which makes the interwiki problem almost disappear.

Re avoiding breakages, I agree. Changes like https://github.com/wikimedia/pywikibot-core/commit/ff69b61805ada9a60995fe3d061c033822f71962 wouldnt be allowed on the 2.0 branch. They would go into master and be part of the 2.1 release which hopefully will have solved these packaging issues.

Having a separate pywikibot-families package makes it easier to release an update to the family files, but how will the user learn of the update and fetch it? Will pywikibot notify them like it does currently in master using 'version.py'? If we also separately release a scripts package, will the user (or version.py) need to monitor four packages for updates: 'pywikibot', 'pywikibot-families', 'pywikibot-scripts', 'pywikibot-scripts-i18n'? John Vandenberg (talk) 18:45, 29 November 2014 (UTC)

No interwiki support in packaged version

As the main issue which requires very up-to-date family files is interwiki support, maybe we can say that the released Pywikibot 2.0 package does not support interwiki.py? i.e. interwiki bot operators need to continue using 'pwb' and a current git clone. This means a few users cant migrate to using the released 2.0 package, but it means the dev team isnt wasting time releasing packages to support them - time better spent making 2.1 work correctly without regularly updating family files. John Vandenberg (talk) 08:07, 1 December 2014 (UTC)