Talk:Wikimedia REST API/Flow

For example, the endpoint to retrieve the mobile-optimized html of the Earth wikipage would be:

https://en.wikipedia.org/api/rest_v1/page/mobile-html/Earth

Hi @Tlietz, In the long term, we are planning to deprecate that endpoint. However, we will be providing a new endpoint with the same functionality. Feel free to use the endpoint in its current form for now, and follow this talk page and the mediawiki-api-announce mailing list to be notified about the new endpoint and timelines for deprecation.

I am trying to get rendered HTML for a bunch of pages. I am pretty sure I am limiting myself to 200 requests per second to page/html API endpoint, and I have User-Agent set, but very quickly I am starting to get 429 responses. Have I misunderstood something about the rate limit for that API endpoint? I am making those 200 requests in parallel though. Could this be a problem? (I use stash=false as well.)

It looks like limit of 100 requests works. (Hm, or not. It still starts returning 429 after some time.) So I tested now with 100 requests per second. I managed to make 45852 requests in 8 minutes and after that they started failing with 429. This is a bit less than 100 requests per second.

So example response:

If you report this error to the Wikimedia System Administrators, please include the details below.

Request from IP via cp3056 cp3056, Varnish XID 528119086

Upstream caches: cp3056 int

Error: 429, Too Many Requests at Wed, 04 May 2022 16:48:01 GMT

I am sad that expected rate limit from hyperswitch does not get reported: https://github.com/wikimedia/hyperswitch/blob/2d4480149ae4d1969eec83063ddec05c9021574d/lib/filters/ratelimit_route.js#L32-L37

Please create a Phabricator ticket and paste the information above so people can check this and help you. Mutante (talk) 17:11, 4 May 2022 (UTC)

I made: https://phabricator.wikimedia.org/T307610

Hi, all! I'm not sure I understand the differences between MediaWiki and Wikimedia REST APIs. I understand that the Wikimedia REST API (not included with MediaWiki) is powered by the RESTbase caching/storing API proxy, making it more appropriate for high-volume use cases. I assume that this is not the case with the MediaWiki REST API.

Why is there a separate MediaWiki API? Is this because having a RESTbase API would be an overkill for most MediaWiki installations?

Thank you!

Hi, Thanks for your question! You are correct: We created the MediaWiki REST API to provide a RESTful interface for anyone using MediaWiki, since running RESTBase would be a complex process for most MediaWiki installations.

I note that we are requested not to make more than 200 requests to the API, but a time interval is not specified. Is this intended to read "per day", or "per hour"? Is there someone to whom requests for a higher access rate might be directed? ~~~~

It actually did say "200 requests/s". I've clarified it, and now it says "200 requests/sec".

I would love to learn about your plans for Service Worker support for the API.

The REST API is primarily targeted at serving the data consumed by increasingly fatter clients, including those taking advantage of ServiceWorkers.

We are also looking into leveraging ServiceWorkers to improve performance and offline support. As part of this effort, we have done some research and prototyping into a) a node environment for running unmodified serviceworker code on behalf of older clients, and b) streaming HTML transformation / composition tooling. We do intend to develop a PWA targeted at mobile web over the next 12 months which might use some of this research, but expect this PWA to be limited to an opt-in beta feature until at least mid 2018.

Who could I ask about the latency of some of the REST endpoints? I see that I can get very recent pageviews data, e.g.

https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/en.wikipedia/all-access/all-agents/hourly/2018031600/2018031800

accessed now, at 2018/03/17 0230 UTC, gives me an hourly pageviews on the English Wikipedia at timestamp "2018031623", so about ~3.5 hours latency, very nice!

In contrast, asking for the daily number of edits via

https://wikimedia.org/api/rest_v1/metrics/edits/aggregate/en.wikipedia/all-editor-types/all-page-types/daily/20180225/20180317

only gives me data up to the end of February. This makes me think the daily datasets are generated only once a month? Who can I ask about this latency, and whether more recent "rest_v1/metrics/edits" data can be obtained? Thanks!

Are there any plans to make the per-user Special:Watchlist available?

It already exists in a different api endpoint (API:Watchlist)?

Most of the endpoints say "deprecated" .. are we supposed use them?

https://en.wikipedia.org/api/rest_v1/

When an end point is deprecated, "it means that it has not been useful to clients and will consequently be removed in the near future." These end points were experimental or unstable before, and have been deprecated as part of a larger clean-up effort in preparation for the 1.0 release.

So no, please do not use those end points any more. They are about to be removed in early May.