Requests for comment/pywikibot 2.0 packaging

Request for comment (RFC)
pywikibot 2.0 packaging
Component	General
Creation date	15 November 2014
Author(s)	John Vandenberg
Document status	in draft

Pywikibot 2.0 is intended to be released as a stable package available via https://pypi.python.org/pypi/pywikibot , pip, and other package distribution systems (rpm, etc)

There is a common held understanding that the Pywikibot library should be designed to be separate from the scripts, and Pywikibot 2.0 codebase mostly achieves that, however there are some packaging complexities which have not been hashed out.

Background

The four issues which complicate packaging are:

cosmetic changes
i18n
scripts
family classes

cosmetic changes

The Pywikibot library depends on the cosmetic changes script in the Page save logic. This uses i18n, and is import as 'scripts.cosmetic_changes'; see sections 'i18n' and 'scripts' below for problems resulting from each.

i18n

The Pywikibot library includes an i18n module, which uses i18n messages are currently extracted from TranslateWiki and placed in a git repository pywikibot/i18n, which is a loaded by git as a submodule of both core and compat. In core, it is loaded into the directory 'scripts/i18n'. Currently the i18n messages are Python data files. There is a pending change to migrate to using JSON data files so that the TranslateWiki team can terminate their Python support.

The Pywikibot library is almost entirely only using English language in user interaction, logging, etc, except it uses i18n for four messages in pagegenerators.py (See https://gerrit.wikimedia.org/r/#/c/166421/2/pywikibot/pagegenerators.py,cm for each in context):

The main usage of the i18n module is by the scripts.

In order to package 'Pywikibot', either the three messages used by pagegenerators.py need to be included, or the dependency on those i18n messages needs to be removed. Patches for both approaches have been submitted; i18n_data and i18n fallbacks. Other options have not yet been explored.

scripts

The Pywikibot scripts currently reside in the same repository as the Pywikibot library, and are collectively placed in a Python package called 'scripts'. There are many instances of a script importing another script, using syntax which isnt supported in Python 3.4. This change 'fixes' that, but continues to call the package 'scripts'.

The name 'scripts' is not an appropriate name for the Pywikibot scripts collection if they were to be packaged on pypi.

There have been numerous scattered discussions about moving the scripts out of the Pywikibot library repository.

family classes

Family classes provide a lot of configuration data about wiki families, needed for pywikibot to operate correctly on those wikis. Most of this information could be dynamically determined at runtime without family classes, and progress is being made on replacing family class functionality with dynamic algorithms. However that effort wont be completed for Pywikibot 2.0. As a result, family classes need to be shipped, and these classes need to change quickly if a wiki site configuration changes.

It would undesirable to issue a new minor release of Pywikibot every time a Wikimedia site configuration change requires a modification to the family file, such as a new language being added to a family.

Proposals

Release 2.0 approach and timetable

As the various packaging approaches depend on code being written and merged, the desired release timetable is an important factor in determine which packaging approach is feasible. If releasing a stable 2.0 package soon is desirable, the packaging method for 2.0 needs to be simplistic, and the development team should then work towards a better packaging method for the release after 2.0.

Below are some options for the 2.0 release timetable, with a description of the approach required to meet that timetable. (RC = Release candidate)

Three months; simplistic packaging only

Pywikibot should be packaged as-is, now, and the packaging method remain stable until 2.0 is released in three months time. Any critical i18n messages and family configuration changes will be released as new minor versions of 2.0; e.g. 2.1, 2.2, etc. Bug fixes will be released as new patch versions; e.g. 2.1.1, 2.1.2, etc

Version 2.0 is only an intermediate step, to give script developers a stable maintained API to migrate to, before a softer migration to the next major version. The development focus should be to fix major bugs in the 2.0 library, and then focus on building the next major release (3.0) to address packaging problems and providing a long term stable release.

2015-01: RC1
2015-02: RC2
2015-03: Release

{Add your name here}

Six months; simple packaging

Pywikibot should be packaged as-is, now, but the packaging should be improved and users should have no expectations that the packaging structure will be 100% stable until April. i.e. Package / module names can change during this time.

The focus should be to fix bugs in the library, but any major packaging problems should be fixed before version 2.0 is released. Version 2.0 actively supported for at least six months, and should be usable for at least a year.

2015-01: beta4 on pypi
2015-02: beta5 on pypi
2015-03: beta6 on pypi
2015-04: RC1
2015-05: RC2
2015-06/07: Release

{Add your name here}

Six months; good packaging

Pywikibot 2.0 betas should be released for testing purposes only until March.

Packaging needs to be designed so that version 2.0 is a stable well maintained release for at least a year, and be usable for years after that in maintenance mode.

2015-01: beta4 on git only
2015-02: beta5 on git only
2015-03: beta6 on pypi
2015-04: RC1
2015-05: RC2
2015-06/07: Release

{Add your name here}

Nine months; extensible packaging

Pywikibot 2.0 should be an extensible framework, with the ability to plug in new Site interfaces, Family classes, scripts, etc., each separately packaged, probably hosted in different repositories with different maintainers.

Pywikibot 2.0 should be the active maintenance and development branch for many years, without breaking changes.

The compat layer should be a separate component, containing CompatSite and CompatPage classes, that is separately maintained as needed by scripts that depend on it, and will see breaking changes whenever any method is no longer needed by any extension distributed by pypi .

2015-02: beta4 on git only
2015-04: beta5 on pypi
2015-06: beta6 on pypi
2015-07: RC1
2015-08: RC2
2015-09: Release

{Add your name here}

2.0 packaging methods

TBD; see talk

Future packaging method

{If you've got ideas or expectations for beyond 2.0, add them here.}