Extension talk:Cargo/Archive January to February 2018

Delimiter for |format=template

Is it possible to set some delimiter when |format=template? It was possible in SMW, and there's some queries I'm trying to switch over to Cargo that were set up like that. --RheingoldRiver (talk) 16:18, 7 January 2018 (UTC)

No - I haven't seen a good reason for it yet. What do you want the delimiter to be? Yaron Koren (talk) 21:28, 7 January 2018 (UTC)

In some cases   (although this one is generally fine to just append to the template, sometimes the template is used in multiple places and I don't always want the leading/trailing space), often •. Sometimes a break between table cells {{BlockBox|break}}, this is the template. There were probably other cases that I used a separator in SMW with template format but these are the ones that come to mind. --RheingoldRiver (talk) 23:06, 7 January 2018 (UTC)

Is the current behaviour to insert a space between result item templates, or is it a newline? Or nothing? (I know I could check, but the couple of places I'm using it it seems to be a newline, but I'm not sure.) Sam Wilson 01:40, 11 January 2018 (UTC)

Okay, that makes sense. I checked in a "delimiter" parameter for the "template" format earlier today - hopefully it will get to you before too long. Sam - by default it doesn't put in anything. Yaron Koren (talk) 03:13, 11 January 2018 (UTC)

Thanks! --RheingoldRiver (talk) 12:19, 11 January 2018 (UTC)

Jobs issue

MW 1.30, Cargo 1.30.

If $wgJobRunRate > 0, records are sometimes stored to DB twice when you save a page. --StasR (talk) 21:21, 12 January 2018 (UTC)

Sorry, what version of Cargo are you using? Yaron Koren (talk) 22:45, 12 January 2018 (UTC)

Git branch REL1_30 — v. 1.4 (1798b60) 13.09.2017 --StasR (talk) 23:03, 12 January 2018 (UTC)

Oh - you should upgrade to version 1.5, which is when the duplicates problem was fixed. Or just use the very latest code. In general, the "REL" branches are not useful for any of my extensions, and possibly not for any non-WMF extensions. Yaron Koren (talk) 23:31, 12 January 2018 (UTC)

Thanks, it helped! --StasR (talk) 00:03, 13 January 2018 (UTC)

We are running 1.5 and still get duplicates. I don't think it is triggered by page save though. For example, I recreated a table about an hour ago because it was full of duplicates. The recreate populated about 450 rows (should have been more like 1500). After an hour, it has 1,330 rows, but many of them are duplicates. Nobody has been editing the pages with dupes, so where are these duplicates coming from? ... Update: Just ran recreate on it again. 430 Rows. Edited my test "Michelle" page, now 867 rows, many dupes but the "Michelle" rows are not duplicated. Since I have been typing, there are now 1266 rows, more duplicates. And for sure zero edits on any page on the wiki.

I have seen this behavior of duplicates appearing in other templates also, but can't figure out what triggers it since sometimes there are none then you come back a few days later and new duplicates have appeared. The dupes are from pages that have not been recently edited.

• This is the table: https://futuramaworldsoftomorrow.gamepedia.com/Special:CargoTables/Actions
• This is the template: https://futuramaworldsoftomorrow.gamepedia.com/Template:Action
• This is one of the pages that inserts data into the Actions table: https://futuramaworldsoftomorrow.gamepedia.com/Michelle

--2angelic1 (talk) 09:52, 12 February 2018 (UTC)

This is very odd - and what makes it more odd is that the "page values" displays don't seem to show any of the values - like here. I'm guessing that these two problems are related, but I don't know how. When recreating, are you populating values into a "replacement table"? If not, I would suggest doing that - maybe it will work better. Yaron Koren (talk) 16:09, 12 February 2018 (UTC)

At the moment you looked at it, the Characters template and table were out of synch due to a template change. Running recreate data fixed what you were seeing there with missing data. It is interesting though, that while the Character template was broken, it was easier and faster to reproduce the dupe issue on the Actions template. It makes me wonder if the dupe issue is something to do with a process erroring out mid-way, causing dupe rows to be added when it is run again. Also, my other thought.. Recreate doesn't actually seem to recreate all the data on the Actions table. It will say jobs="0" but all the rows won't be there. I wonder if that process breaking mid-way has any impact. And yes, I usually do a replacement table, check it out and then switch. For the Actions table, I have tried both ways to see if it affected dupe issue and doesn't seem to matter. (2angelic1 (talk) 6:03, 14 February 2018 (UTC)

Let me also add that there's no reason to have all those #if calls in the #cargo_store call in that template, as far as I can tell. I doubt that's the cause of any of these problems, but it would at least simplify things to remove them. Yaron Koren (talk) 21:00, 13 February 2018 (UTC)

Stripped out all but one if statement in the Actions template (it was an if/then and made sense). Not going to impact issue with dupes, but agree those weren't needed there. (2angelic1 (talk)6:07, 14 February 2018 (UTC)

Alright. This is odd - though less odd than before, thankfully. Is it correct that the "Actions" table is currently the only one with duplicates? If so, could it be due to the recent-ish changes to the template - not any specific changes, just the fact that it was changed. Could you try recreating that table one more time? Perhaps this time will work better. Yaron Koren (talk) 14:08, 15 February 2018 (UTC)

This duplicates issue has been happening since at least November, and happens to other tables. I just used Actions as an example since it seems to do it the most frequently so easiest to see/reproduce. I recreated yesterday and see dupes back in there today. Other wiki admins on the gamepedia platform have the same issue and some have written some ugly custom queries to work around this problem. We think it is either an issue with the source or the configuration on the gamepedia servers. They have a ticket in and have also been looking into it but no result so far. Thanks for your time. (2angelic1 (talk) 12:56, 16 February 2018 (UTC)

Well, I thought it was just "Actions" because I skimmed through the other tables and couldn't see any duplicates. As far as I know, for other wikis (on Gamepedia and elsewhere) that are having problems with duplicates, it's just because they're using versions of Cargo older than 1.5 - though I could be wrong about that. Clearly there's still a problem here, though I can't figure out why. By the way, I see that you're now using Cargo 1.6 - do you know when that upgrade was done? Yaron Koren (talk) 16:51, 16 February 2018 (UTC)

All Gamepedia wikis are on the same version. Not sure when they upgraded to 1.6 though. In the futurama wiki, you can also see some dupes currently in the Unlock_Materials and Goal_Parts tables. I periodically go in and recreate data on tables with dupes to try and clean them up. The only problem with it is it doesn't always recreate all the data. Not sure if related to the dupes issue or not but I think that issue with the recreate process has been reported elsewhere in here. (2angelic1 (talk) 20:35, 16 February 2018 (UTC)

Oh, good to know all Gamepedia wikis are running the same versions - I didn't know that. Anyway, based on what you just said, my guess is that you are running a recreate before the previous recreate has finished, which is leading to the duplicate rows. Why the duplicate-checking code in Cargo is not catching it, I don't know, but that's my theory - so I would recommend just waiting longer between one recreate and the next. Yaron Koren (talk) 21:30, 16 February 2018 (UTC)

I can confirm that running a recreate prior to the previous one finishing definitely causes a ton of dupes (or at least did in the past). And if the earlier recreate didn't populate all rows initially so I blank edited all pages, then recreated relatively soon after that, it still made a bunch of dupe rows, so even though it was done populating in that everything showed up, Cargo didn't think it was done. The tables I was recreating were pretty big (by our standards at least), ~3k (non-dupe) rows or so. I haven't reproduced this in a while (probably just because I'm not rebuilding tables that much anymore) but I can try to reproduce when I get a chance, though it might be a bit complicated because I don't want to break stuff on our live wiki. --RheingoldRiver (talk) 23:06, 16 February 2018 (UTC)

Yes, recreating data and running recreate again (or editing pages using that template) before it is done seems to make dupes worse. However, this isn't the source of the dupes problem. Running recreate was an attempt to clear the dupes. Instead of helping it seems to make the problem worse. It is also usually weeks between me running a recreate to clear dupes. Every wiki seems to have a different strategy to deal with the dupes. I've been trying to recreate to clear them. Others have been doing group by queries so the page results just don't show the dupes. (2angelic1 (talk) 11:32, 19 February 2018 (UTC)

Well, I don't know, then. Gamepedia wikis are still the only ones I know of where data duplication is a problem, post-version 1.5. And the code in Cargo that tries to prevent storing duplicate rows seems pretty foolproof to me. Could it be that there are some local changes to the Cargo code on Gamepedia that are preventing this code from working correctly? That's my new theory. Yaron Koren (talk) 00:03, 20 February 2018 (UTC)

Just in case it helps someone troubleshooting this issue. I just watched the duplicate problem happen. It occurred when I null edited a page. Instead of replacing the existing rows, it added a new set of rows. This page calls the same template 25 times and another template 1 time. It was the template with 25 items to update that duplicated the rows. Doing another null edit right after updated the same data correctly, clearing the dupes. So it happens often but not every time. Same thing with recreate. Adds some rows then quits, but not consistent as to which row it glitches on and quits. (2angelic1 (talk) 12:01, 1 March 2018 (UTC)

There have been some improvements to the Cargo storage code in the last month that may solve this issue - I don't know. I'm hoping to release a new version of Cargo soon. Yaron Koren (talk) 04:33, 2 March 2018 (UTC)

Underscore inconsistency Part II

Previously I reported there is inconsistency with use of underscores in queries and templates. For some reason underscores must be replaced with spaces when referring to fields in templates. (Of course, if this is changed, many templates must be redone, as they now use the workaround of space instead of underscore).

It looks like I came across another issue related to this one (unless "type" is another banned SQL keyword?). In for instance Enchantment_poe1, try to click on one of the "next 250" and such links, and an error message pops up (notice the lack of underscore):

Table "Enchantment poe1" not found in Cargo database.

–Pangaearocks (talk) 09:08, 13 January 2018 (UTC)

Wow - it's surprising that no one saw, or at least reported, that error until now. My guess is that it's in part because the "view next pages" functionality in Special:Drilldown is rarely used, since the page is meant to winnow down options to a manageable size. Anyway, I just checked in a fix for the problem; sorry about that. Yaron Koren (talk) 14:01, 15 January 2018 (UTC)

Missing tail end of record set when calling same #cargo_store template multiple times from the same article

I have certain pages for which I want to store multiple rows of data in the same table for a single page, I'm doing this by calling the same #cargo_declare/#cargo_store template multiple times from the page, with different variable values. While this does work, I have noticed that random pages often experience missing rows. Upon examining this, I've noticed that it's always the tail end of the records that go missing. Consider a page trying to store 5 rows with the values 1-5 to some table, using a template, and five separate calls like this:

{{Template|value=1}}
{{Template|value=2}}
{{Template|value=3}}
{{Template|value=4}}
{{Template|value=5}}

When this works, five records are created like

_pageName	value
Page	1
Page	2
Page	3
Page	4
Page	5

When rows are missing, it's always an uninterrupted set from the end, leaving the table in a state like for example

_pageName	value
Page	1
Page	2

Different rebuilds of the same data results in different pages missing a different amount of records (from the end), so I'm ruling out malformed input, as a template call that sometimes works does not always work. I have experimented with recreating the data into a replacement table, checking for missing rows, null-editing all pages calling the template, checking again for missing rows, null-editing again, etc. and each set of null-edits has a chance to add some of the missing rows, but not always, and not necessarily all of them (causing a page to still be missing the tail end of records, but a lower amount of them).

Do you have any insight into what might be causing this behavior? Worth mentioning is that I'm using a Scribunto module to calculate the values which are to be set, and using the Lua frame:expandTemplate{ title = title, args = table } for each call, which may or may not prevent the Cargo data from being properly stored on page re-cache, as pages with missing records do not get them added after the incomplete replacement table has been switched in, even through null-edits or scheduled page cache refreshes. SaschB (talk) 05:09, 14 January 2018 (UTC)

My guess is that the jobs are timing out before completion, possibly due in part to the complexity of these storage calls. Is your "job run rate" set to something higher than 1? If so, I would try bringing it down to 1 (or even less). Or, if you have command-line access, you could try running the cargoRecreateData.php script, instead of recreating from the web interface - that should work better. Yaron Koren (talk) 14:04, 15 January 2018 (UTC)

This is on Gamepedia, so the admin/config is handled by other people, I asked them about the job run rate, and it's apparently set to 1. A separate issue is that, while newly created pages (with less complex call structures) are picked up fine, any edits to pages which already have some data stored do not seem to update the live tables. This means I have to use the web interface to recreate tables (and subsequently null-edit all category members, as it were) after any edit I want immediately usable. I believe these recreates might not be strictly necessary, but in many cases I can't wait 24 hours for the tables to refresh, and null-edits or purges of pages do not update live tables - only replacement tables (additionally, null-edits to pages which store data in replacement tables delete that page's rows from the live "read-only" version of that table, which is yet another problem).

All of this could probably be solved if a null-edit or purge of a single page would properly update the live table, since this would eliminate the need for recreating tables after every edit. Is this something you recognize as an issue, and might know what it could be caused by? (Under SMW, purging a page did properly update the values) SaschB (talk) 19:24, 15 January 2018 (UTC)

While a replacement table exists, a "live" table is read-only, as you note. (Or at least, it's intended to be that way.) Given that, there is (or is supposed to be) no way to modify the "live" tables that still have a replacement table - it has nothing to do with caching or purging. Maybe that's an error in design - I wasn't taking into account the possibility that recreating a table could become a very lengthy, complex process, as it is for you, due to (I think) your combination of complex template calls and lack of command-line access. Yaron Koren (talk) 13:31, 16 January 2018 (UTC)

They're not strictly read-only though, as any edits to pages storing data in them deletes rows from them (I consider deletion a write-operation). Even so, the live tables are effectively read-only even while there are no active replacement tables, as page edits aren't stored in them unless the table is recreated. SaschB (talk) 16:32, 17 January 2018 (UTC)

Oh... if the "live" tables are getting modified, that sounds like a bug. Yaron Koren (talk) 20:50, 17 January 2018 (UTC)

This was indeed a bug, and I believe it's fixed now. Yaron Koren (talk) 22:02, 21 January 2018 (UTC)

Query continuation

Hi !

I've started to use the Cargo API, but I'm facing an issue. I want to make a request that would yield massively over 500 results, but the server limits the results to the first 500 first elements. Is there a way to "continue" a request, to get the next results ?

For some reason, I never added an "offset" parameter to #cargo_query, or to the "cargoquery" API action. That was probably a mistake. There might be another way to do this, though: you can use the page Special:ViewData as an API (its "csv" and "json" formats basically function like an API), and it does allow an "offset=" query string parameter. Yaron Koren (talk) 02:18, 21 January 2018 (UTC)

Option to store integers without commas

Is there any way to do this, or if not could it be added? We would like to be able to use #expr: on results queried that have type Integer, and also store years (2017, 2018, etc) as type Integer and have them print without commas. Right now we just #replace all of the commas with nothing but it would be nice to be able to skip this step. --RheingoldRiver (talk) 22:44, 20 January 2018 (UTC)

To be clear: the values are stored without commas; the commas are added when the values are displayed. I didn't know that #expr failed if there were commas in the numbers - that's too bad. You may be able to get rid of them just by putting "CONCAT()" around the field name within the #cargo_query "fields=" parameter, so it's "CONCAT(MyIntField)" instead of just "MyIntField". But as far as I know, there's no reason to store years as type Integer - I think it's always better to store them as type Date, which will get rid of the commas problem there. Yaron Koren (talk) 02:24, 21 January 2018 (UTC)

We have a very weird reason to store years as Integer type, which is that our 2011 and 2012 seasons in competitive League of Legends esports were called Season 1 and Season 2, and then after that it switched to "2013 Season" etc. So seasons are 1, 2, 2013, 2014, etc. That said we could probably just store as String. I'll check if CONCAT() works to get around the other problem, thanks. --RheingoldRiver (talk) 22:57, 21 January 2018 (UTC)

Case Sensitivity in HOLDS

Sorry, one more question - when we move our game stats from SMW to Cargo in the future, I want to join a table that has a column listing all the different names player has played as to the table of game data and use HOLDS to get all of the games that the player played in, regardless of name at the time. Is there any way to make this query case-sensitive? We don't always have disambiguation links for players that had the same name but with different casing (for example ILLusion and illuSioN), especially when the player with unusual casing is very obscure. --RheingoldRiver (talk) 22:51, 23 January 2018 (UTC)

No problem, keep the questions coming. I don't think you can do that with "HOLDS" directly. Here you can see the longer query that a "HOLDS" query corresponds to - I think you'll have to go with the longer query, and put something like "LOWER()" around both sides of the comparison. Yaron Koren (talk) 14:54, 24 January 2018 (UTC)

Sorry, that's the opposite of what I want - ILLusion and illuSioN should be distinct entries so that if I query one I only get his games and none from the other. Is that how it works by default? --RheingoldRiver (talk) 19:44, 24 January 2018 (UTC)

Oh... never mind. Yes, HOLDS is case-sensitive. Yaron Koren (talk) 19:53, 24 January 2018 (UTC)

Are you sure? Illviljan tried here and it seems to be insensitive. - https://pathofexile.gamepedia.com/User:Illviljan/cargo#Case_sensitivity_in_HOLDS --RheingoldRiver (talk) 20:42, 24 January 2018 (UTC)

Oops again! Somehow I forgot that the '=' operator in MySQL is case-insensitive. I just looked it up, and thankfully there's an easy solution: change "HOLDS" to "HOLDS BINARY" (or "=" to "= BINARY"). Yaron Koren (talk) 22:10, 24 January 2018 (UTC)

Yep that works! Thanks --RheingoldRiver (talk) 23:58, 24 January 2018 (UTC)

Feature Request: Parameter for adding styles/classes to display formats

Would it be possible to add a parameter for styling the results of a cargo query? I understand that this can be done with templates but since there are already a bunch of nice display formats available it would be great to be able to specify |class=red to make colored versions of the tables, for instance.

Right, there's no "class" parameter for any of the formats. You can accomplish the same thing, though, by wrapping the query in a "div" or "span" tag, like '<div class="red">'. Then the CSS needs to look like "div.red table ...". Hopefully this is only a little bit of a hack. Yaron Koren (talk) 15:21, 25 January 2018 (UTC)

Attaching to multiple tables

Hi! Is there a specific reason why you can only ATTACH to exactly ONE table within a template? While having multiple instances of #cargo_attach (on different tables each), only the last one is computed and applied during data recreation... --149.243.232.3 14:44, 26 January 2018 (UTC)

That sounds like a bug - I'll have to look into it. Sorry about that. Yaron Koren (talk) 17:17, 26 January 2018 (UTC)

False positives from HOLDS queries

Using a query like this sometimes produces results where {{{1|}}} in fact isn't part of FocusHeroes. (For example [1] returning [2])

{{#cargo_query:tables=SummoningFocuses
|fields=_pageName,StartDate,EndDate
|where=FocusHeroes HOLDS "{{{1|}}}"
|group by=_pageName
|order by=StartDate ASC
|format=template
|template=Template:HeroFocusListFormat
|default=* This hero has not appeared on any summoning focus.
}}

Am I doing something wrong? SaschB (talk) 03:56, 28 January 2018 (UTC)

The query looks correct - something strange is happening with the storage. I would recommend recreating that table - if you still get errors like that even after recreating it, this may be a bug that still exists in Cargo. Yaron Koren (talk) 05:00, 28 January 2018 (UTC)

Recreating the table seems to have worked, but if whatever caused it to happen is still in effect, it's bound to reappear in the future, in which case I'll come back to this post. SaschB (talk) 12:39, 28 January 2018 (UTC)

That's good. Yes, please let me know if you see this problem again. Now I'm wondering about a different problem, though, which is that the table you just recreated, SummoningFocuses, has a lot of duplicates - and it shouldn't have any, given that the Cargo code now includes a check to avoid duplication. I can't think of any reason why that's happening, and I couldn't replicate the problem on my system (using some of your data). Do you know if the Cargo code on your wiki exactly matches what's in the main Cargo code (at least, at some relatively recent point in time)? Or are there any custom changes? Yaron Koren (talk) 17:03, 28 January 2018 (UTC)

Couldn't tell you if there are custom changes, but you have some contact at Gamepedia, right? You should be able to ask them about it. I did recreate this table in my usual fashion: recreate with replacement table -> bot-null-edit all category members calling the cargo_store template -> switch in replacement table. This usually causes duplicates for the ~50% of rows the "recreate table" command manages to pick up straight away, since they are also null-edited a minute later, causing the duplicates. SaschB (talk) 17:46, 28 January 2018 (UTC)

Aha - the fact that these are bot edits may have something to do with it, since bot edits seem to cause problems for Cargo, at least in some systems. Why are you doing these null bot edits? If it's to help with recreating the data, they may not be necessary. Yaron Koren (talk) 23:05, 28 January 2018 (UTC)

Yeah, the bot null-edits are to ensure all pages are present in the cargo table "immediately", so that the replacement table can be safely switched in without causing missing data for an unknown period of time. The bot null-edits do cause duplicates, but at least they seem to totally prevent missing items. A "Recreate using replacement table" I performed just now seems to have stopped at 58 records ([3]) while the current live table contains 90 unique _pageName records. SaschB (talk) 15:50, 29 January 2018 (UTC)

Well, it's at 81 rows now, although it looks like the jobs have all run (you can always check at this link). I don't know why 9 of the pages didn't get their data stored. In any case, I think you can safely call that null-edit bot now if you want, without fear of leading to duplicate data. Yaron Koren (talk) 17:42, 29 January 2018 (UTC)

That's a useful link I didn't know about, I'll keep that in mind when tinkering/troubleshooting SaschB (talk) 20:17, 30 January 2018 (UTC)

Haven't done any null-editing on the pages in this table/category yet, a couple of new pages have been added, so the total should now be 91 or 92. Table is up to 89 records, 72 hours after being recreated. SaschB (talk) 18:45, 31 January 2018 (UTC)

I guess there were remaining jobs after all, then, even though that statistics API link said there were none. Well, it's great that the table is getting fully populated, although it's too bad that it's taking so long. This is the kind of thing that makes me glad that the "replacement table" option exists, though. Yaron Koren (talk) 20:05, 31 January 2018 (UTC)

Most of the time, when adding new data, we can't wait several days, as information is often announced/made available mere hours before being needed (by users), so I guess I'll keep batch-null-editing to help it along, even if it causes duplicates. SaschB (talk) 08:28, 1 February 2018 (UTC)

Alright. In any case, you can always create a replacement table the "right way" afterwards, as you did here. Yaron Koren (talk) 15:41, 1 February 2018 (UTC)

I'll probably skip that step, as there's no simple way of knowing when the replacement table is finished (the one in this example wasn't finished after 72 hours, and I don't know when/if it would have been finished without manual intervention). I'd rather have a complete table full of duplicates than an incomplete table without duplicates. SaschB (talk) 17:21, 1 February 2018 (UTC)

A replacement table lets you have both, in a sense, until it gets filled completely and can be switched in... Yaron Koren (talk) 18:24, 1 February 2018 (UTC)

Well, yes, I see your point, but with new data being added with our frequency, we often have to rebuild the tables once every two weeks - so keeping track of a table for one week in order for it to lack duplicates (but work the same) for one week doesn't make sense to me. I know we're not supposed to have to recreate the tables when data is added/edited, and some of these issues have been solved, but I'm not sure if all of them have been (yet). SaschB (talk) 21:57, 1 February 2018 (UTC)

Yeah, on that note - sorry to belabor the discussion, but I didn't understand what you meant about new data being added. Are you just talking about more data, without any changes to the data structure? If so, what's the benefit of recreating the table? Yaron Koren (talk) 22:08, 1 February 2018 (UTC)

I am just talking about new data (INSERT or UPDATE), but with previous issues of pages not updating the live cargo tables correctly/immediately upon page edits, my workaround to ensure all new data was stored was the previous procedure of recreate -> replacement -> null-edit -> switch in. As of recently, it seems the storage is a bit more robust, so hopefully I won't have to in the future. If I have to, however, that's what I'll do. SaschB (talk) 12:12, 3 February 2018 (UTC)

It happened again: [4] shows [5] as a result, even though the value isn't there. Also [6]/[7]+[8] and [9]/[10] SaschB (talk) 16:59, 3 February 2018 (UTC)

That's too bad that this storage problem is still there in the latest code, but of course it's good to know that it's there. I looked into this, and I think I figured out what was causing it. I just checked in a fix - hopefully it'll get to you soon, and hopefully it really does fix the problem. Yaron Koren (talk) 05:26, 5 February 2018 (UTC)

CSV output only exports 100 rows

The CSV output format only outputs the first 100 rows. Is there any way to get it to output more? --pcj (talk) 18:56, 29 January 2018 (UTC)

You can set the number with "limit=" - hopefully you can set it high enough to include all the rows. Yaron Koren (talk) 19:12, 29 January 2018 (UTC)

Error with Special:CargoTables after deleting attached templates

Hey there. I recently deleted a bunch of templates that were attached to a table and subsequently visiting the Special:CargoTables page gives me this:

[94d2634e1606751386c8fc50] /a3/index.php?title=Special:CargoTables TypeError from line 145 of /home/sites/1b/e/e73edcb322/public_html/a3/includes/linker/LinkRenderer.php: Argument 1 passed to MediaWiki\Linker\LinkRenderer::makeLink() must implement interface MediaWiki\Linker\LinkTarget, null given, called in /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/includes/CargoUtils.php on line 971

Backtrace:

#0 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/includes/CargoUtils.php(971): MediaWiki\Linker\LinkRenderer->makeLink(NULL, NULL, array, array, array)
#1 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/specials/CargoTables.php(298): CargoUtils::makeLink(MediaWiki\Linker\LinkRenderer, NULL)
#2 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/specials/CargoTables.php(353): CargoTables->tableTemplatesText(string, array, array)
#3 /home/sites/1b/e/e73edcb322/public_html/a3/extensions/Cargo/specials/CargoTables.php(28): CargoTables->displayListOfTables()
#4 /home/sites/1b/e/e73edcb322/public_html/a3/includes/specialpage/SpecialPage.php(522): CargoTables->execute(NULL)
#5 /home/sites/1b/e/e73edcb322/public_html/a3/includes/specialpage/SpecialPageFactory.php(576): SpecialPage->run(NULL)
#6 /home/sites/1b/e/e73edcb322/public_html/a3/includes/MediaWiki.php(283): SpecialPageFactory::executePath(Title, RequestContext)
#7 /home/sites/1b/e/e73edcb322/public_html/a3/includes/MediaWiki.php(851): MediaWiki->performRequest()
#8 /home/sites/1b/e/e73edcb322/public_html/a3/includes/MediaWiki.php(512): MediaWiki->main()
#9 /home/sites/1b/e/e73edcb322/public_html/a3/index.php(43): MediaWiki->run()
#10 {main}

I'm not sure if the deletion of the templates was actually the cause but I'm at a loss re: how to fix it. Viewing the table pages themselves seems to be fine and nothing else is borked as far as I can tell.

Rurikawa (talk) 13:46, 1 February 2018 (UTC)

Very odd - looking through the code, and your backtrace, it looks like the problem is that one of the "attaching" templates that you deleted did not get its "attached to a table" page property deleted from the page_props table - so the Cargo code sees the property, goes to display a link to that template, then crashes because the template doesn't exist. I didn't know that could happen. Anyway, I just checked in what is hopefully a fix, so that the code checks whether a page is null before trying to link to it. Yaron Koren (talk) 15:29, 1 February 2018 (UTC)

That fixed it! Thank you for the quick response. Rurikawa (talk) 19:14, 1 February 2018 (UTC)

Data not updating.

Hello,

I have installed Cargo, and added a couple of templates (http://crewsgenealogy.com/wiki/index.php?title=Template:Infobox_person). I belive I have added the code correctly, cargo doens't produce any errors when I set up the tables, but it doesn't seem to be saving any of the information. I have tried the maintenance script cargoRecreateData.php which ran without errors, but still nothing. I have several pages with the template, so I would think something would be saved. If it makes any difference it seems to be stuck with the message "Note: One or more of these tables are currently being populated, via the job queue." even after I ran the script.

Any suggestions?

I see the #cargo_declare which defines the table itself, but I can't see a #cargo_store, which defines which data to store in the table. Might be as simple as that? SaschB (talk) 12:18, 3 February 2018 (UTC)

Example:

{{#cargo_store:_table=Person | honorific_prefix = {{{honorific prefix|{{{honorific_prefix|{{{honorific-prefix|{{{pre-nominals|}}}}}}}}}}}}| name = {{PAGENAMEBASE}}}}

Automatic |no html added when query used in page forms?

I have this query in a form, and the query works but it seems like it's adding a |no html to it. This is how the same query looks embedded in a page. Is this a Cargo issue, and if so any way to fix it? Thanks! --RheingoldRiver (talk) 17:35, 3 February 2018 (UTC)

There are a number of complicated aspects to that query you have - maybe the most relevant one is that the query is itself calling a template, in order to make the table display more customized. If it's not too much work, could you try testing to see if a simple table display of the query, instead of that template display, results in that same error? If not, there may be some workarounds (though ideally your current query should work as well). Yaron Koren (talk) 14:36, 4 February 2018 (UTC)

That table actually has no fields that directly have markup stored into them, but I made this instead and the markup is showing properly. We also have a number of query forms in SMW with relatively complicated "line" templates such as this one that I was hoping to be able to switch over to Cargo in the next couple months. --RheingoldRiver (talk) 21:56, 4 February 2018 (UTC)

Something that may be related is that on occasion we will see similar behavior on an actual wikipage, i.e. it looks like this after saving. As soon as another user opens the page, it displays normally [edit - actually this may not always be the case, after a comment from someone on another Gamepedia wiki that they had the issue persist on another user viewing the page]. This is the page that screencap is from, though it can happen on a number of pages, and this problem isn't unique to the LoL wiki, a bunch of Gamepedia wikis have experienced the same thing. In the case of that page I changed "Smeb" to "Smeb2" and back, repeatedly. Whether the bug happens or not is relatively inconsistent, but I was playing around with user rights on my bot account, and it seemed to happen more often when I had Admin than when I didn't. --RheingoldRiver (talk) 22:49, 4 February 2018 (UTC)

It sounds like you're saying that the Special:RunQuery bug only happens when the "template" format is used in the query. Which is still bad, but not as bad as if happened for every query. Until this bug is actually fixed, I can think of two possible workarounds: (1) switch to the "table" format, and replicate as much of the logic as possible using MySQL functions, and (2) do the whole query display in a Lua module. (I suppose there's a third option, which is to use SMW for all these queries, but I'm leaving that aside for now. :) ) Would either of these be an option?

As for the second issue - this is the first I've heard of this. My only thought is that this is related to whether the page is being cached or not; although I can't see the error on that page in (I think) either state. If you can figure out some way to reproduce it consistently, that would be very helpful - though that's probably obvious. Yaron Koren (talk) 01:20, 5 February 2018 (UTC)

Yeah that should probably be an option for this template, though we definitely have some that I wanted to convert to Cargo where it wouldn't be possible. I'll let you know if I see any (Cargo-related) bugs with it when trying to change it to just a table query.

I tried for about an hour to find a combination of user rights that led to the problem happening consistently, and sadly I couldn't. This error has been around for a while, but it's always been super intermittent; and it's never happened when I'm logged in as myself (in Firefox fwiw, while my bot is in Chrome). And since our last issue with saving pages was a conflict with another Gamepedia extension it's also possible that that's what this is. AFAIK this is also only happening when |format=template (maybe suggesting the two are related?), but nearly all of my queries are |format=template so that's not necessarily the case. I'll ask around and see if anyone had the same issue occur with another format. --RheingoldRiver (talk) 07:57, 5 February 2018 (UTC)

Oh also, let me know if you want Editor on our wiki ever since we have FlaggedRevs enabled. --RheingoldRiver (talk) 07:59, 5 February 2018 (UTC)

Preventing cargo_store on a transcluded page

I'm trying to use Cargo for a blog, and am using cargo_query to transclude blog-post pages onto a front page (using format=template, and the template then transcludes the full page with {{:{{{_pageName}}}}}).

The problem I've got is that I don't want the cargo_store directives to be processed on the main blog page (i.e. where the most recent 10 posts are listed). To this end, I tried wrapping the cargo_store in a conditional statement:

{{#ifeq: | {{PAGENAME}} | Blog ||
{{#cargo_store: _table=blog_posts
| date = {{{date|}}}
| location = {{{location|}}}
}} }}

But this for some reason this means that the cargo_store is never processed, on any page. Running cargoRecreateData.php reports the correct number of pages that call the template, but no rows are stored in the table.

Does anyone have a suggestion of what I'm doing wrong? Thanks! Sam Wilson 02:40, 10 February 2018 (UTC)

Are you 100% sure you have no typo in the cargo store vs the cargo declare field names? If there's a single field in the declare that doesn't exactly match a field in the store it won't save any row. --RheingoldRiver (talk) 12:43, 10 February 2018 (UTC)

Yep, good point though. It does work though as soon as I remove the wrapping {{#ifeq: | {{PAGENAME}} | Blog || ... }}, so I'm assuming there's something odd going on with the PAGENAME magic word. Anyway, I should probably be looking at doing this a different way, and perhaps storing the text of the blog posts as template parameters or maybe using _pageData table. I'll keep experimenting! Sam Wilson 03:20, 11 February 2018 (UTC)

The syntax of the ifeq is wrong in your example on this page: {{#ifeq: | {{PAGENAME}} | Blog || ... }} has an extra pipe at the start, and should be {{#ifeq: {{PAGENAME}} | Blog || ... }}. That might be it? The example you provided compares PAGENAME to [blank], evaluates to false, skips the true value "Blog", to render the false value [blank]. SaschB (talk) 08:20, 11 February 2018 (UTC)

Gosh, I feel very silly! :-) That was indeed the problem. Thank you. I think I'd stared at it for far too long. Sam Wilson 00:49, 12 February 2018 (UTC)

Let me just add that it would be slightly simpler to use "format=embedded" instead, which would hopefully have the same output. Yaron Koren (talk) 02:35, 12 February 2018 (UTC)

Page edits breaking Lua queries?

Hi, me again! Example page, invokes a Lua module which performs a Cargo query:

  local ctables = 'Heroes'
  local cfields = '_pageName,WeaponType,MoveType,Rarity'
  local cquery = { groupBy="_pageName", limit="500" }
  local heroQueryResult = cargo.query(ctables, cfields, cquery)

and then sorts through the results, displaying them in a custom table.

Whenever one of the pages returned as a result by this query is edited, it disappears from the page, presumably due to no longer being returned by the query. The edited page still shows all the data in the Cargo table, but it disappears from the module-generated table on this page, and others like it.

Missing unit, not yet missing unit. Both from the same cell (Shuriken column, 3-4* row).

Any ideas why? SaschB (talk) 07:24, 11 February 2018 (UTC)

My only thought is that maybe these newly-saved pages are now exceeding the query limit, because they've moved to the end of the results. I know you set a limit of 500, which should be more than enough, but maybe it didn't "take", for one reason or another. Is that possible? Yaron Koren (talk) 18:52, 11 February 2018 (UTC)

I'm not sure I follow, the table itself only has 225 records total, at the moment, so the limit shouldn't be a problem, right? Are you saying I should edit the local cquery = { groupBy="_pageName", limit="500" } part to something else? SaschB (talk) 09:58, 12 February 2018 (UTC)

I guess so - I'm saying, perhaps the "limit" value is not getting sent correctly. It doesn't seem that likely, but it's my only theory. Yaron Koren (talk) 15:42, 12 February 2018 (UTC)

To be clear, it doesn't have to be an edit to the #cargo_store-calling template, just adding some text to a different part of the page also results in their data disappearing from the lua-generated table, but not from the Cargo tables. Ever since this edit to the "Lyn" article, "Lyn" has been missing from the page I linked initially. Do page edits like this also cause the records to move to the bottom of the SQL tables? SaschB (talk) 16:23, 12 February 2018 (UTC)

Yes. Every edit to a page, no matter how small, wipes out the Cargo data for the page and recreates it. Yaron Koren (talk) 16:30, 12 February 2018 (UTC)

Okay, I figured it out. A few weeks ago, an anonymous user made an invisible (but actual) edit to one of the deeper templates used to generate the values stored in the Cargo table. This change only updates the values once each page is edited, but doesn't produce visible changes to the value, however it does break the Lua code. Simply put, this isn't indicative of any Cargo issue at all, false alarm! SaschB (talk) 17:06, 12 February 2018 (UTC)

phew That's great to hear! Yaron Koren (talk) 18:48, 12 February 2018 (UTC)

API improvements

Hey :) I was just playing around with the Cargo api and noticed a few things I'd like to bring to your attention:

• Field names starting with an underscore (_pageName, _pageID, ...) only work if a display name is set (e.g. fields=_pageName=pageName).
• The results key is always title (Something like the row id might be more useful here).
• Some way to get a list of all tables and all fields in a table might be a nice addition to the api.
• Not limited to the api, but is there some way to use fields=* in Cargo?

--178.2.47.251 20:09, 22 February 2018 (UTC)

Hi - my responses:

• Are you sure about that? If so, what version of Cargo are you running?
• I don't understand this one.
• That's an interesting idea; what would it be useful for?
• Unfortunately, no, but it would be nice to have. Yaron Koren (talk) 17:11, 23 February 2018 (UTC)

Thanks for your answer!

• Yeah I'm sure I just tested it again (See &fields=_pageName and &fields=_pageName=someName). I'm on version 1.6.
• I meant that the json output always has this format (with "title" being static instead of something specific like the row id):

{
    "cargoquery": [
        {
            "title": {
                "someField": "someData",
                "someOtherField": "someOtherData",
            }
        },
    ]
}

• Having a way to get all cell names would make it possible to simulate fields=* and easily get all data in the table without having to specify the field names in the script and having to change it whenever the table changes. For the table names I'm not quite sure how useful it is and just felt like it was strange there was no way to get them through the api. --178.2.47.251 21:38, 23 February 2018 (UTC)
• Another thing I forgot about yesterday is the lack of an offset option which would be useful due to the results limit.

• Oh, I didn't realize you were talking about the API here too. That sounds like a bug; I'll have to look into it.
• If you want nicer-looking JSON, your best bet is probably to use the "json" result format, instead of the API - it functions as its own sort of API.
• That's true - being able to get the list of field names would be helpful, especially when calling from Lua or the like.
• Yes, I agree that an "offset" parameter would be good too. Yaron Koren (talk) 21:53, 25 February 2018 (UTC)

Some of these issues are now resolved in the new version 1.7: having a "_" at the beginning of a field alias leads to an error message in API queries, and an "offset" parameter was added. Yaron Koren (talk) 19:10, 9 March 2018 (UTC)

Data types with incorrect capitalization are being stored as varchar

If I create a "float" field then Cargo is creating the SQL column as varchar(300) instead of float. "Float" operates correctly. Please add some case-insensitivity handling to the data types. --pcj (talk) 18:38, 23 February 2018 (UTC)

QOL Improvements for page values page

Hey, two related requests about the ?action=pagevalues view. 1 - could you add a table of contents at the top of the page listing all tables that are included on the page? And 2, could each heading link to the table? (Or these could be combined as a list at the top with anchor & table link listed for each table.) Not super important but would be a nice QOL thing. Thanks! --RheingoldRiver (talk) 01:30, 24 February 2018 (UTC)