Manual talk:Job queue

The following discussion has been transferred from Meta-Wiki.
Any user names refer to users of that site, who are not necessarily users of MediaWiki.org (even if they share the same username).

Meaning of the numbers

Is 6,999 a long or a short job queue? Are there rough estimates how long it takes to work through a job length of length X? I guess there is no perfect answer, but some examples would help already (like what the maximal length was and how long it took to work through that). Kusma 21:40, 4 June 2006 (UTC)

Per default, one page request to the wiki will take one item out of the queue. So, it'll take 6,999 page requests to empty it - how long that takes depends on how many visitors the wiki has. On the english wikipedia, this will be a few minutes, i guess. -- Duesentrieb 23:51, 4 June 2006 (UTC)

Is the queue common for all WMF projects, or do each have a queue of its own? \Mike(z) 16:59, 1 November 2007 (UTC)

One job per request?

This seems not to be true, using version 1.6.7. I have got a private wiki and I have tested the jobs queue by accessing my Special:Statistics page and watching the queue. It seems to me, that every request runs several sub-requests, maybe while the css are being loaded (MonoBook.php):

 @import "/w/index.php?title=MediaWiki:Common.css&action=raw&ctype=text/css&smaxage=2678400";

I had to use something like $wgJobRunRate = 0.1; to have only one job runned each time I pressed F5 to reload.

--jsimlo 13:48, 21 July 2006 (UTC)

How to empty job queue?

In Thai Wikipedia, the number of the queue now are about 14,000 and never decreases. Anyone know how to empty the job queue. Please see at w:th:Special:Statistics. --Manop 21:53, 7 August 2006 (UTC)

Run maintenance/runJobs.php --68.142.14.71 14:57, 31 August 2006 (UTC)

It doesn't work to me neither http://th.wikipedia.org/w/runJobs.php, http://th.wikipedia.org/w/maintenance/runJobs.php nor http://th.wikipedia.org/maintenance/runJobs.php --Manop 21:20, 1 September 2006 (UTC)

Because you are not (and probably never will be :) allowed to do it. Running such scripts is reserved for the system administrators only. --jsimlo 21:31, 1 September 2006 (UTC)

Thank you, the queue is empty now --Manop 17:08, 6 September 2006 (UTC)

changing noinclude text on a template

On Commons we have some very very highly used templates like commons:template:GFDL.

I just added some interwiki links to the noinclude section and then the job queue is over a million. :o

I would have thought changes only inside a noinclude shouldn't really add jobs to the jobqueue? Or is that a special case not worth implementing separately?

--pfctdayelise 06:16, 25 October 2006 (UTC)

I think this sort of thing is starting to be a real cornern. At time of writing, the job queue is about eight hundred thousand, and has been for a while. I have no idea what in particular was responsible (not me, I promise!), but one suspects this is more likely to be due to a small numbers of edits to very high use templates, than a huge number of separate edits. Perhaps some sort of priority queue would also be better, so that the larger jobs go to the back, and swamp small ones less. If it's heuristically possible to guess which are more likely to have a significant effect, that would be handy too. Alai 05:15, 11 November 2006 (UTC)

I have a related question. Say in noinclude part of template A there is a call to template B. Now, if I change template B, will all pages including A be added to the job queue? --Paul Pogonyshev 21:07, 6 December 2006 (UTC)

A record?

I was going to ask if 146,716 was the longest the job queue had ever been, as I've not seen it above 35,000 ish before. Then it leapt up to 315,451 - is there something odd going on? To almost quote Withnail (or I): "I demand to have more graphs!" 193.134.170.35 14:19, 13 December 2006 (UTC)

That's quite impressive/alarming, all right. It seemed to be regularly in six figures for a while, but aside from alarming graphs, what would be interesting to see is some analysis of why the job queue is as high as it on such occasions. Some sort of per-edit cost attribution, ideally, especially if it's something preventable, such as people making "documentation tweaks" to the transcluded code of high-use templates, and other silliness. Alai 05:53, 14 December 2006 (UTC)

Usually, because someone edits a template included in half the wiki, or something similar. I've seen the en.wikipedia job queue rise to 900,000 items on occasion, usually as the result of several users editing high-profile templates at the same time. Titoxd^(?!?) 05:10, 16 January 2007 (UTC)

Again, it's hit 141,290. Why are there no records for this? Graphs really would be fantastic, there must be some way to do this? 129.215.149.97 11:12, 25 February 2007 (UTC)

0.9M is... impressive. I understand that's the likely cause, but unless we can find out which templates are being edited to cause this, it doesn't really help us, does it? (If we knew which, we might be able to, say, upgrade their protection, dope-slap the people editing them unduly, restructure them to not be such severe "single points of failure", etc, etc.) Alai 04:44, 14 March 2007 (UTC)

It was about 700,000 last night, and has been over 300,000 all day today. That's either a lot of template edits, or some templates that are absurdly heavily used. If there are any devs hanging around here, could they comment on the feasibility about adding a "job queue by top ten templates" breakdown, or something along those line? Alai 21:13, 18 March 2007 (UTC)

Over 2000k

Right now it's over 2000k, and there's no sign its decreasing. This is amazing because it's more than the number of articles (I know the job queue includes user pages, etc., but still ... ). Some sort of breakdown from how these jobs initiated would be great. CMummert 16:19, 4 April 2007 (UTC)

See also Wikipedia:Village pump (technical)#Job queue. - Jc37 23:14, 4 April 2007 (UTC)

You mean Wikipedia:Village pump (technical)/Archive#Job_queue, revision 124277233 --193.11.177.69 23:25, 19 September 2007 (UTC)

As for how it can be larger than the number of articles, per haps some of the transclusions are to some of the same pages. For example, in the discussion I linked to above, there are some user pages which have (due to template transclusion) categories of Wikipedian programmer, User bas, User bas-1. If I remove the parent cats from the template, then two separate actions will be listed in the queue (I presume), just for that page alone. I am, of course, guessing : ) - Jc37 23:14, 4 April 2007 (UTC)

Currently 2,302,658 ... seems awful big. -- ProveIt 23:52, 4 April 2007 (UTC)

How long does it take for the system to work through this sort of job queue? My bot's starting to report high rates of outdated category information. --67.185.172.158 02:15, 7 April 2007 (UTC)

It can take forever - if you have no traffic on your wiki. I.e. it depends on the amount of traffic you have on your wiki. --Sebastian.Dietrich 08:12, 31 January 2008 (UTC)

I'm going for a new record. Right now I got it up to 3.7M. I heard reports that some wikis have hit 4M already! (Commons and en.wp) Hint: Lots and lots of edits to your most used templates in a very short period of time is the best way to do it. Another idea is to edit a bunch of templates than use Special:Nuke to revert them all and then repeat as needed. 75.4.160.23 10:57, 17 March 2008 (UTC)

Let me see if I have this right

It's good to get your facts straight before calling design into question. So let me see if I understand this system:

• Edit a template and save it and the system puts all the necessary page updates (Pages which transclude the template) into a job queue.

• Mediaiwki's job queue empties based upon surfing activity. "By default, each time a request runs, one job is taken from the job queue and executed."

This means that, the more pages viewed by a visitor, the more jobs are called from the queue. One view one job! So Mediawiki adds load to high load period!

Oh we can tune it down with that exciting variable, but that is the way it's designed, right? It depends on surfers visiting pages for the job to be taken off the shelf in the dusty cupboard called a queue.

Now, if I'm wrong then my comments are also wrong. Take that as a given:

• Any sane designer would then run a cron job that looked at load factors and emptied the queue fast in low load times and slowed down in high load times.

• No visitors = no queue emptying, but that actually does not matter at all, because they don't need to see anything if they aren't there.

I have several embryo wikis using this software, solely because of the user interface. Last night I added a 3,000 (approx) job item to the job queue as an experiment. I have a template that is on 50% of my pages and I edited it. I wanted to see what happened because I did not believe the way the queue worked.

My server was lazing about drinking pina coladas and generally getting a tan. There were minimal wiki visitors (I chose deliberately my lowest load period), and it didn't even break sweat, it just asked for more sun cream and a refill of its glass. It should have taken the low load opportunity to empty the queue like a rocket!

Oh no. 18 hours!

So, where do I raise this?

No point going to Bugzilla, but this is the most unusual, cockamamie (is that how you spell it) piece of alleged design I've seen since I started as a programmer in Algol in 1975!

The justification will be a pseudo justification of "We need this to run on servers where the admin has no access to create cron jobs" or "But it doesn't just run in a *IX environment", but that just isn't an answer, it's a justification for a massive kluge.

Now, let's just suppose I'm wrong. Then I apologise for my current comments and ask "Why is it so slow? Why doesn't it take advantage of low load periods?"

And if I am right? Guys, some designer needs to fall on his sword.

Ah yes. I don;t have an account here, sorry. 91.84.170.194 16:40, 29 November 2007 (UTC)

Seems as if you are right. I've changed a template yesterday evening --> job queue length was ~ 3.000. And after ~ 9hours (with no traffic an the serer just idling around) still ~ 3.000.

Job queues hold jobs and jobs are asynchron. Jobs should therefore be always executed in a background task. One can always assign this task a higher priority when the queue gets too long...

--Sebastian.Dietrich 08:10, 31 January 2008 (UTC)

The issue is not that the jobs are held in a queue. That is good design. The issue is the idiotic design that says "empty the queue fastest when the site is at its busiest!". This is imbecilic design. In a commercial operation I would be retraining the idiot who designed it. 82.152.248.89 11:50, 2 April 2008 (UTC)

End of content from meta.wikimedia.org.
Note that the above conversation may have been edited or added to since the transfer. If in doubt, check the edit history.

Sane designers have better things to do than implement a load-aware scheduling system that requires no installation effort and works in safe mode PHP. Use cron, fool. -- Tim Starling 05:26, 21 December 2008 (UTC)

Semi-protection

Could somebody Semi-protect this page, because here's often ip-vandalism -Fujnky 05:41, 18 June 2008 (UTC)

Spanish

Hi. I've tried to reach this page from the Spanish version of Wikipedia and all I got was a redirect page to this one. where a message tells me not to edit it. It seems like something wrong happened while translating these contents into Spanish. Is it possible for me to translate it, show it to you and later copy it into the corresponding Spanish page? Thanks. --Dalton2 11:20, 4 July 2008 (UTC)

Hebrew

Do you want a translation to Hebrew of this page? I will be more then happy to help. Contact me at my talk page. 82.81.44.31 19:29, 7 December 2008 (UTC)