Talk:Requests for comment/Minimalistic MW API Client Lib Specification

Comments by BJorsch (Anomie)

I have no objection to someone doing this, or something like it. But I don't want to be expected to maintain it myself, not least because I'm not very familiar with Python. BJorsch (WMF) (talk) 16:57, 3 June 2015 (UTC)

Not a problem, I think we have plenty of python devs around. This lib should simply show "good practices" from the API-devs perspective, as described in the docs. --Yurik (talk) 16:59, 3 June 2015 (UTC)

token

Getting a token using different versions of the API involves three different API calls. See API:Token and API:Tokens (action) for two of them.

Would this API module handle all three versions? John Vandenberg (talk) 05:49, 4 June 2015 (UTC)

That's a good question - I feel we should not support legacy api token approach in this library, and instead let the middleware such as pywikibot to handle all of the older cases. --Yurik (talk) 15:13, 9 June 2015 (UTC)

Yurik, wow; need some clarity on that one. Are you proposing that this library:

1. only supports the new, v1.24+ method? or,
2. supports the v1.20+ method and not support the even older method?

Either way is OK, as pywikibot can (and should) handle the older cases rather than making this new API layer more complex, but I worry that v1.24+ would reduce uptake of this new API layer, as most clients/middleware that perform write actions will already support v1.20+ login, and wont easily shift to this new API library if it means they only support v1.24+.

(also please see continuation section below, as that could be a deal-breaker for pywikibot). John Vandenberg (talk) 02:24, 10 June 2015 (UTC)

@John Vandenberg: , same as i wrote below for continuation - lets only support the most recent API (as relevant to Wikipedia) as of this moment, but structure it in such a way so as to allow for the middleware (e.g. pywikibot) to alter the security mechanisms, and inject older style authentication/cookies/etc. --Yurik (talk) 23:53, 29 July 2015 (UTC)

console logger?

Doesn't that make it more like a minimal bot framework than a client library? Also, did you consider mwclient? It's supposed to be quite minimal, and you'r welcome to join forces improving it if you like :) Danmichaelo (talk) 20:49, 4 June 2015 (UTC)

The example ConsoleLog looks like it is a very basic implementation of converting InfoObject's into messages shown to the user. It is useful for lazy programmers who are using this api layer directly in a quick hack, without using a proper client that would handle the InfoObject's properly.

I dont mind this layer having a utility functions for that purpose, but I do think this is bad approach. Callers should use logging.basicConfig if they want a basic console logger, and this new api layer should provide a utility function to accept a InfoObject and translate it into logging calls. That promotes correct python usage, however that is an implementation detail.

mwclient is way too large for what Yuri & others are asking for. They expressly do not want a package that has a configuration file, with all the code to handle that. They also do not want this layer implementing api calls (e.g.allpages, etc) except those explicitly included in this specification, e.g. login, token.

My limited understanding of their desires is that they also do not want this new api layer to do any site initialisation, or handling of backwards compatibility. (mwclient and pywikibot both do that, fetching siteinfo,userinfo,etc). I am still unclear about this part, as implementing 'token' has implications (see section above). John Vandenberg (talk) 09:21, 5 June 2015 (UTC)

I've never heard of another python library that provides a logging interface like this. I think it should be dropped. We should move discussions of tokens to an appropriate thread. --EpochFail (talk) 12:58, 7 June 2015 (UTC)

We need some interface through which the library could notify of various logging events, with the default going to console. Suggestions welcome. --Yurik (talk) 15:16, 9 June 2015 (UTC)

How about we use logging? --Halfak (WMF) (talk) 20:51, 5 August 2015 (UTC)

Upon re-reading, I think that I missed the point of the InfoObject class. It seems that this would be used to report the notices ("warnings") that accompany an API response. It seems like this name is problematic. It seems that the API docs just call these things "warnings", so I'm not sure why we wouldn't just call them the same thing: Warning. It seems to me that we should have a way to access the warnings associated with a response, but it's not clear to me that we ought to wrap them in some data structure. --Halfak (WMF) (talk) 21:13, 5 August 2015 (UTC)

@John Vandenberg: : Just to clarify: mwclient does not use a configuration file. It ships with a helper file that can be used to create mwclient instances from a configuration file, but I've never used it myself. Danmichaelo (talk) 13:49, 7 August 2015 (UTC)

Re logging lib - sure, works for me - as long as it will be easy for the pywikibot and such to totally take over it. One thing to consider - would a warning or an info log message imply that overriding framework might want to take a different course of actions? Like pywikibot might need to instruct the lib to retry query with a different arguments?

Re custom object - api currently returns simple warning strings, but sometime in the future, it will start returning structured data - e.g. localized strings, substrings, etc. Do we want to provide for the forward migration with a wrapper object here? --Yurik (talk) 21:20, 5 August 2015 (UTC)

Costs of maintenance

I don't think that this addresses real-world issues. I think that it is much more laborious to port a tool to use this library than to watch the API announcements and implement the (very few) breaking changes that occur from time to time.

In addition, while the current breaking change in continuation may be solvable by such a library, I assume most breaking changes will, well, break workflows.

This doesn't mean that there shouldn't be some responsibility on the part of WMF to only deploy when it is sure for example that Pywikibot doesn't break. With MediaWiki and Pywikibot both using Phabricator for task management and talk of setting up Pywikibot as a Jenkins test, this should be easily doable.

Also, it may be useful for Pywikibot to outfactor some functions. As others have pointed out, this should take into account the existing libraries in the field. --Tim Landscheidt 01:50, 5 June 2015 (UTC)

I tend to agree wrt Pywikibot. The Pywikibot folk at the Lyon meeting were more interested in documenting what 'core' looks like now to dispel some rather large myths about its unsuitability for some tasks, and continuing to lighten pywikibot of stuff inherited from before the MW API even existed, and/or move towards a more componentised system (i.e. the Family class). Then we can have a better discussion about future architecture with all stakeholders having a common understanding about what 'Pywikibot' is. A new API layer is not about preventing breakages in Pywikibot; we've already got enough testing occurring that we've prevented any API breakages hitting production 'core' bots in the last 12 months, and most of these API breakages have not been 'announced', and often they are unintentional. Where we have a weakness is that we only run all tests on new checkins for pywikibot, which means API breakages may slip through if merges dont occur on a deploy days or people dont triage build breakages (which does happen when the builds are red for long periods).

However I do agree with user:Yurik and user:Halfak (WMF) that our ecosystem needs a reference implementation in Python, which would be useful for building new smallish tools, or building new clients or even libraries which provide a rich layer to a narrow part of the API. It may be that pywikibot functionality is moved into new libraries that use this new simple API, while the more complex parts of pywikibot continue to use its own API layer. John Vandenberg (talk) 09:48, 5 June 2015 (UTC)

On top of John Vandenberg's comments, I'd like to add that, as a developer of an API library that is well used -- but not at pywikibot's level, I think this does address real-world issues. It seems common that people who work in the pywikibot space tend to be blinded to the needs of python tools devs who work without pywikibot -- which honestly makes sense, but leads to these sort of "why don't we all just use pywikibot" complaints. Generally, I think that there was strong pressure towards a more modular strategy expressed at the Lyon hackathon. Pywikibot is an operating system that was designed to support a specific type of wiki tool. If you're not building a pywikibot bot, you probably don't need or want pywikibot's monolith.

However, I still think that Tim Landscheidt raised a legitimate limitation of working at the basic level. If we were to encode generalizations into this level that let us handle API changes (beyond simple things like continuation), then we'd be moving beyond the basic level. That's OK. If you want to build a layer on top of this basic library that handles API changes (e.g. pywikibot core, mediawiki-utilities, mwclient, etc.), then this basic library will at least make sure that your middleware library doesn't need to re-implement login, continuation, errors/warnings and sessions. As one of those middleware devs, I'm looking forward to not needing to think about those things. :) --EpochFail (talk) 13:26, 7 June 2015 (UTC)

Why explicitly handle tokens

I don't remember discussing this. I don't see a good justification for why tokens should be included along-side login(). And if we are going to have a facility for gathering tokens, why not provide wrappers for methods that require tokens (e.g. edit(), login(), move())? Can't we just call() action=query&meta=tokens? --EpochFail (talk) 13:09, 7 June 2015 (UTC)

Note that I started a section about this above. John Vandenberg (talk) 13:35, 7 June 2015 (UTC)

Silly me. Sorry about that. Let's continue the discussing in the earlier thread. --EpochFail (talk) 14:08, 7 June 2015 (UTC)

continuation

In the section above titles 'Costs of maintenance', user:EpochFail mentions that he thinks this library should handle continuation, if I understand correctly. Like token, this is another example of a feature that is version specific. Should this new layer handle both methods of continuation? If so, does that mean this library also does API version detection? John Vandenberg (talk) 06:31, 8 June 2015 (UTC)

You have mentioned me here, but it seems that this is discussed in the proposal as well. See Requests_for_comment/Minimalistic_MW_API_Client_Lib_Specification#query(**kwargs). --EpochFail (talk) 13:35, 8 June 2015 (UTC)

I am hoping to hear from more voices here before agreeing on a design, or even starting to do a proper design, especially on the contentious aspects. So far everyone is happy with this new library being MediaWiki 1.21+ only. John Vandenberg (talk) 15:21, 8 June 2015 (UTC)

@Yurik: , can you confirm that you believe this new layer should be 1.21+ only? John Vandenberg (talk) 17:38, 29 July 2015 (UTC)

@John Vandenberg: , apologies for not seeing this earlier. Yes, I feel the minimalistic API should only support the 1.21+ API for continuation (and possibly introduce future compatibility changes as the MW code changes). That said, I feel that minimalistic API should expose the underlying Request object, so that if the bigger framework is capable of working with the older MW, it could reuse some of the relevant code in the minimalistic api - e.g. ability to make api calls, proper parameter encoding (list->pipe-separated string, etc) - all those things that have been the same from the start, without using the continuation or the newer security/cookies api calls (if applicable). --Yurik (talk) 23:49, 29 July 2015 (UTC)