Extension talk:Cargo/Archive August to September 2017

Treatment of blank date fields

When I don't enter a date for a date field it defaults to the current date when it is displayed in the template page, but there is blank when it is displayed in a dynamic table query. This is only a minor issue but something I thought I would mention for you. I'm not really sure whether it's a Page Forms or Cargo thing. Jonathan3 (talk) 00:08, 5 August 2017 (UTC)

I'm not sure it's either. What's the template page? Yaron Koren (talk) 01:38, 7 August 2017 (UTC)

You're right. It's neither. The template uses {{#time: j/n/y | {{{Date field|}}} }} and {{#time: j/n/y | }} defaults to today's date (e.g. now it displays 7/8/17). I guess I need to check for an empty date field, or just not leave it blank... Jonathan3 (talk) 20:00, 7 August 2017 (UTC)

Cargo Boolean should be 0/1 or yes/no by definition - how is this done, if database column type created is boolen?

I am not able to figure out the problem with Cargo fields of type Boolean. So if I declare field of type Boolean, when data is recreated, a column of type Boolean is created in the database table (I use PostgreSQL). How does this correlate to Cargo/Storing data definition: "Boolean - holds a Boolean value, whose value should be 1 or 0, or 'yes' or 'no'"? When defining a form, I declare a field by using cargo field=<cargoField> and form correctly displays checkbox for Boolean field. When data is stored, the Boolean value in the database is also correctly stored as true or false. However, I have issues when viewing or drilling down the table via Special pages:Cargo tables options. The table shows only Yes values for all Boolean fields and similar issue occurs with Drill-down - only None and Yes counts, no No counts. How to fix this? Blazm (talk) 19:00, 24 August 2017 (UTC)

That's odd. What versions of Cargo and MediaWiki are you using? Yaron Koren (talk) 20:41, 24 August 2017 (UTC)

Here is info about my system and samples of code for template and form:

Software Versions:

• OS: Alpine Server Linux
• Mediawiki: 1.29.0
• PHP: 7.1.7
• PostgreSQL: 9.6.4
• SMW: 2.5.4
• Page Forms: 4.1 (bb3328e)
• Cargo: 1.3.1

Template:

<noinclude>
{{#cargo_declare:_table=itequip
|ipv4=String
|dhcp=Boolean
}}
</noinclude>
<includeonly>
{{#cargo_store:_table=itequip|ipv4={{{IP Address|}}}|dhcp={{{DHCP|}}} }}
{| 
! {{PAGENAME}}
|-
! IP Address
| {{{IP Address|}}}
|-
! DHCP
| {{{DHCP|}}}
|}
</includeonly>

Form:

<noinclude>
{{#forminput:form=ITEquipmentFormEnter}}
</noinclude>
<includeonly>
{{{for template|ITEquipmentTemplate}}}
{| class="formtable"
! IP Address: 
| {{{field|IP Address|mandatory|placeholder=192.168.1.x|cargo field=ipv4}}}
|-
! DHCP: 
| {{{field|DHCP|cargo field=dhcp}}}
|}
{{{end template}}}
{{{standard input|save}}}
</includeonly>

In the Postgre SQL a table cargo_itequip is created which among other contains columns ipv4 of type character varying(300) and dhcp of type boolean.

If I check the Cargo table, I see: NOTE: DHCP for the SERVER in the database is set to FALSE

Page	SERVER
ipv4	192.168.1.254
dhcp	Yes

If I check Drilldown:

• ipv4:
  • None(x) * 192.168.1.254(1)
• dhcp:
  • None(x) * Yes(9)

Blazm (talk) 7:04, 25 August 2017 (UTC)

What do you mean by "check the Cargo table"? And what is the value of that field in the "SERVER" page's wikitext? Yaron Koren (talk) 13:55, 25 August 2017 (UTC)

"check the Cargo table" > Special pages;Cargo;Cargo tables and then use either View or Drilldown. "SERVER" page will show the correct information - DHCP: No. Blazm (talk) 19:40, 25 August 2017 (UTC)

Alright, thanks. So Cargo is able to store boolean values correctly, but not read them again. I'm guessing that this is a PostgreSQL issue, but I don't know how exactly - and unfortunately there's no easy way for me to test directly with PostgreSQL. I do have a theory, though. In the drilldown interface, do those 9 "Yes" values represent 9 pages with "Yes", or 9 pages with "No", or a combination? Yaron Koren (talk) 11:54, 27 August 2017 (UTC)

Template/doc example stored in Cargo table, and other questions

I'm a new admin at the Pillars of Eternity Wiki, a reasonably big gaming wiki. It's running SMW, but recently we've installed Cargo to see about migrating. For now I've made a template using Cargo, which has its data here. If you look at that page, you'll see that there are two rows of a creature called "Example", which is from the Template page, and Template/doc page. Naturally I don't want these stored in the table, but I'd also like to have a proper example in the documentation, so people can see what it looks like. I hope more people will contribute because of it.

Firstly, how can I solve this, while keeping the the example in the documentation, but preferably without having to use e.g.:
{{#ifeq: {{NAMESPACENUMBER}} | 0 | ...... }}, which I think also means I need to use {{!}} ahead of every line in the #cargo_store.

Secondly, since I assume this is practice in many wikis, would it be possible to alter the extension so that it doesn't store data from the Template namespace unless we specifically ask it to? Unless it's possible to avoid this behaviour with a parameter I don't know about ofc.

Then I have a few more questions, related more to formatting.

1. I've made some examples of output format, and like the table format best, as it fits design on the site, and it's easy to read. However, is it possible to make these tables sortable? I don't like the extra fields in the dynamic table output format, like search and page numbers, and it also appears to be on the slow side from what I've seen so far.
2. Is it possible to put more than one variable in one cell in such a table? For instance, it would be preferable to be able to put all 6 attributes in one cell, to save width space.
3. Three of the variables are of the List (*) of Page type. This seems to work well, I can click on the page in Drilldown. But in the tables I just get e.g. ". . . . . " (see here for instance). What must I do to get these values to show up in tables? Do I need a sub-query for instance? Could you show me an example of how to do this?

Thank you for an excellent extension. I hope it's possible to migrate the whole Wiki to it, or at least more parts in due time. I may have more questions as I test and learn more about it :) Pangaearocks (talk) 17:44, 29 August 2017 (UTC)

Hi - I haven't been asked about disabling #cargo_store before, but it certainly makes sense. I'll have to think about the best way to implement that in the code; for now, the #ifeq approach you suggested seems like the best bet. I don't think you would need to replace pipes in the call.

1. It's not possible to make the regular table format sortable, though that does make sense as a feature, especially for larger tables - like more than 20 rows.

2. I think you could put multiple fields together by having one field be a "CONCAT" call, like "CONCAT(FieldA, ', ', FieldB, ', ', ...)=MyCompoundField".

3. I think the issue with those list fields is that the values are already in bracketed links, and Cargo expects the page names to just be strings. So you should either remove those square brackets, or change the type from Page to String (or maybe even Wikitext).

I'm glad you like Cargo so far! Yaron Koren (talk) 20:33, 29 August 2017 (UTC)

In relation to question 1, it's not very neat but you can use the "template" output format to make sortable tables.

{|class="cargoTable sortable"
!A
!B
!C
|-
{{#cargo_query:
tables=...
|fields=A,B,C
|format= template
|template=...
|named args=yes
}}
|}

And in the template page:

<includeonly>
|-
|{{{A|}}}
|{{{B|}}}
|{{{C|}}}
</includeonly>

You could modify this to answer question 2 also.

For question 3, I'll guess and say I think the best answer is just to remove the [[ and ]], as that should allow the Page type to work properly - and stop the [[ and ]] from appearing in the Drilldown page too. If you use String any [[ and ]] would show as plain text in queries too, I think. Jonathan3 (talk) 22:27, 29 August 2017 (UTC)

Thanks a lot for the swift answers. Then I hope this will be possible in future versions of Cargo :) I'd love a sortable table output format, because we'll have at least 100 rows on multiple CargoTables (including the bestiary, which I'm currently working on).

Haven't had time to test the code you replied with yet, I broke my leg yesterday so things haven't exactly gone quite to plan. But I will later. However, I have another query, about Redirects. Some of the creature names in Pillars of Eternity have funky names, and one of them is the Mênpŵgra. Naturally that required a redirect from the 'clean' name. However, this means that I have two versions of Mênpŵgra in the CargoTable (from here and here). How can I best solve this?

As for the [[ and ]], I use those because I want the page links in each creature page, in the infobox, so people can quickly go to them. But maybe List (*) of Wikitext would work better? (instead of Page). The asterisk is so that the wiki automatically turns them into bulletpoints (ul). See an example here for instance. Pangaearocks (talk) 20:08, 30 August 2017 (UTC)

I could do with sortable tables as an option too, but until Cargo changes the code above produces the same result.

Instead of a redirect how about the DISPLAYTITLE magic word? Though I think that only has an effect when viewing the page itself (not in the Cargo query). Or find/write an extension to replace every instance of the plain word with the fancy version.

The Page type will create hyperlinks so there's no need to use square brackets.

Hope your leg ends up all right too... Jonathan3 (talk) 06:49, 31 August 2017 (UTC)

Pangaearocks - I'm sorry too to hear about your leg! I think there's one error in what Jonathan3 wrote, which is that, as you note, links indeed do need to be added manually for display within the page itself. I would recommend using #arraymap (defined in the Page Forms extension) for that - it lets you automatically add links around every value within the field. In fact, you can use #arraymap to add the bullet points as well - so that the template field just looks like "MyField=A, B, C", while it gets displayed as a set of bulleted links. I urge you to check out #arraymap if you haven't heard of it. Yaron Koren (talk) 17:18, 31 August 2017 (UTC)

Thanks for explaining that about the Page type. I hope this is relevant enough to belong here - could you set out the differences between String and Page types? Jonathan3 (talk) 00:44, 1 September 2017 (UTC)

The difference comes when they're displayed in a query - Page values get linked. Yaron Koren (talk) 01:40, 1 September 2017 (UTC)

┌───────────────────────┘
Thanks for the help so far, and the well-wishes :) After some trial and error, I managed to convert the template and use #arraymap to create the bulletlists in the pages. Ended up using {{#arraymap:{{{ability|}}}|,|x|* [[x]]|\n}}, which worked quite well.

Now I wonder more about formatting again, and how best to approach it. It looks a bit tricky to format each column in the table generated by cargo_query the way I'd like, and make the table itself sortable, so would it therefore be better to make a template with a table, and let output format from the queries be template? Do you have an example of how this might be done, besides what Jonathan3 wrote above? Unfortunately I'm not all that savvy with code, although I'm trying to learn. Would I be able to use normal template syntax, like {{{variable}}} to format the table, or am I mixing up cargo syntax with normal wiki syntax too much? Here is the current test page: https://pillarsofeternity.gamepedia.com/User:Pangaearocks/Antelopes

Btw, I couldn't get HOLDS LIKE to work. A long SQL error message was thrown back about syntax (can it only be used for lists?). But LIKE worked, which is what I'm currently using. Pangaearocks (talk) 02:27, 3 September 2017 (UTC)

A follow-up to this, as I sort-of got it working through Jonathan3's example. As he said, it's not pretty, but it works. One of the motivations of potentially changing to Cargo, however, was to easier fetch this data and format it. This way isn't dynamic, and with data split between template and page. It would have been far preferable if we could do something similar, but get variables AND values from cargo, then format it in a table, and send it back. That way, we might be able to make one 'formatting' template, which then adapted to amount of columns. I hope something akin to this will be possible in the future. Or better yet, making the table display format sortable, and making it easier to customise formatting of it (just making it sortable would be a great help, though). For now, the splitting up approach is a decent enough workaround, albeit not ideal. Pangaearocks (talk) 23:33, 3 September 2017 (UTC)

Yes, "HOLDS LIKE" is only for list fields. And yes, adding a "sortable" parameter to the "table" format seems like a good idea. Yaron Koren (talk) 01:22, 4 September 2017 (UTC)

Three database glitches (duplicate rows, rows not being added, table giving errors)

First off, I absolutely love Cargo -- thanks so much for your hard work on it.

I am, however, experiencing some problems with it on a wiki that I'm trying to set up. The three problems are:

• One of my tables is returning errors when I try to view it via Special:CargoTables. You can see this here: [1] This happens no matter how many times I delete/recreate the table. The template that creates this table is located at: [2]. However, this problem is mostly unrelated to the following two (more troublesome) problems:
• I am also having the "duplicate rows" problem that others have reported. It is particularly noticeable in the Trophies table, which is created by the Trophy template. (I will explain how to reproducably make a duplicate row shortly)
• I have made a bot that creates some pages on the wiki automatically. The first time those pages are created, their entries are (correctly) added to the Trophies table (linked above). However, if I then "Recreate Data", those rows will *not* be added back into the table until a human edits them.

For example, you can test this weird behaviour as follows: right now, the Trophies table is missing many rows (from pages created by my bot). If you do a null edit on this page, it will be added to the Trophies table (likely twice). If you then "Recreate Data" for the Trophies table, that double row will be corrected, but it will *remove* numerous other rows (e.g., the row with _pageID equal to Game/dark-souls/trophy/1 will be removed).

Thanks for your help, and let me know if there's any other information that I can provide that would help. NathanielJ (talk) 18:59, 1 September 2017 (UTC)

For the first issue - if you add "$wgShowExceptionDetails = true;" to LocalSettings.php, it should display a more helpful error message. For the issue of pages getting ignored during "Recreate data" - do you have a job run rate higher than 1? If so, that might be the problem. In either case, running cargoRecreateData.php from the command line might fix it. Yaron Koren (talk) 22:16, 1 September 2017 (UTC)

Thanks for your help Yaron. I set $wgShowExceptionDetails = true; so that a more detailed error message appears here. Unfortunately, I do not understand what it means though.

For the recreate data issue, the job run rate is set at exactly 1. I ran cargoRecreateData.php, but that didn't help -- the Trophies table only has data for the 4 pages that use that template that have been edited by a human, but not the dozens of pages that use that template that were created by my bot. NathanielJ (talk) 22:35, 1 September 2017 (UTC)

As a follow-up to the previous message, I ran the maintenance script refreshLinks.php and then ran cargoRecreateData.php again, and that fixed the Trophies table. NathanielJ (talk) 22:53, 1 September 2017 (UTC)

Ah - that error message is very helpful, although it doesn't look like it. That was a bug with Cargo's handling of fields whose name contains a number, which was fixed on May 17, soon after version 1.3.1 came out. I would recommend upgrading to the very latest Cargo code, if you have Git on your system. If not, I'm actually planning to release the next version of Cargo, 1.4, in the next week (now that the hierarchy feature has been added), so you can just upgrade then.

As for your call to refreshLinks.php - that's very good to know too. I had just assumed that it was a coincidence that the bot-created pages were not being handled, but now it makes sense: the bot-created pages did not have an entry in the database linking them to the template that they call, and Cargo uses those links to know which pages need to have their data stored. Now I'm curious: what does your bot look like? Specifically, I'd like to know the line or lines of your script in which the actual edit call is done.

Finally, there's the data duplication problem - I'm less worried about that one, because I'm actually planning a fix for it. Though I admit that it's annoying at the moment. Yaron Koren (talk) 13:44, 3 September 2017 (UTC)

Thanks again for the reply. I'll just wait for Cargo 1.4 to fix the Games database issue (the site isn't public yet so I'm in no rush). The bot that I'm using runs in PHP via Chris-G's botclasses.php script. Then the line of code that creates a wiki page just looks like (for example) $wiki->edit('Game/game-name-here', 'Content of the wiki page.', 'Summary for the revision history.'); If that's not specific enough, I can e-mail the exact code to you. NathanielJ (talk) 21:13, 3 September 2017 (UTC)

Okay, that's very good to know, and that's specific enough. I still have no idea why the links are not getting refreshed when the edit is called from a script, but clearly this is an issue; I'll have to look into it. Yaron Koren (talk) 01:26, 4 September 2017 (UTC)

If you're reading this - I finally tried to replicate that problem with API edits, using botclasses.php, but couldn't do it. Though I should note that this seems like a MediaWiki problem, not a Cargo one, if I understand it correctly - if a page edit that adds a template call does not result in the link between the page and the template getting stored somewhere, that is somehow a problem with the running of MediaWiki itself. Yaron Koren (talk) 13:34, 6 October 2017 (UTC)

Thanks for the update -- I agree with you that it's likely a MediaWiki issue, not a Cargo one (I'm also having some trouble getting bot-created pages to populate in categories, for example). NathanielJ (talk) 15:02, 12 October 2017 (UTC)

Cargo 1.4 changes

In a previous comment I see that there are plans to release an update to Cargo, Cargo 1.4, in the near future. In that update, will the fixes/changes to doubling up of data/rows and making the table display format sortable be included? I must admit it's rather frustrating to see the cargoTable all out of shape again, after I had to make a change to the template behind it, and all connected pages. Roughly half the creatures (rows) are now doubled up, and the other half missing, which messes up all creature pages to a bigger or lesser extend since they are querying for data as well now. Therefore it would be fantastic if this issue could be solved in the 1.4 update. --Pangaearocks (talk) 09:57, 5 September 2017 (UTC)

There's nothing in the new version that deals with either duplicate or missing rows, unfortunately. I'm more concerned about the missing rows - have any more appeared yet? If not, I would suggest running another recreate - my guess is that you may have run two or more in a row, and that they overlapped, i.e. one started before the previous one finished. Yaron Koren (talk) 14:39, 5 September 2017 (UTC)

I had a small hope and had to ask ;-) No new rows had been added, so I followed your advice and recreated the table (did indeed do that twice last time, as the first attempt only resulted in few rows, maybe 15%). Not everything showed up now either, but 96 out of 126 is a better ratio, and I hope the rest shows up over the coming hours, or at least a day. Still some doubling up, but not nearly as much as last time. I asked the Gamepedia staff if we had access to runJobs.php somewhere, but all those things are locked out (probably due to security), so I'll have to wait until the behind-the-scenes stuff do their thing. Pangaearocks (talk) 16:33, 5 September 2017 (UTC)

Underscore inconsistency?

Am working on adapting display format from cargoTable via a template, and noticed something odd. Some of the variables have underscores in the name. Naturally I used underscore when listing fields in the cargo_query, and in the template (with named args set). Yet for some reason these values did not show up. However, when I keep the underscore in fields, but remove it (for space) in the template, the values correctly show up in the displayed table. Had noticed earlier that underscores didn't show for these variables in the CargoTable (view CargoTables), but didn't think much of it. However, maybe there is an oddity in the code? It would make more sense if we had to use undescore in the variable when working with them in templates too, as afterall the variable name actually contains underscore. In my case, some of them are damage_lh, dr_base, dr_burn.

Cargo query, with a little wonky code to get the table sortable.

{|class="wikitable sortable"
! class="unsortable" | Image 
! Name 
! Level 
! class="unsortable" | Endurance 
! class="unsortable" | Damage 
! class="unsortable" | Attributes 
! class="unsortable" | Defenses 
! class="unsortable" | DR
! class="unsortable" | Abilities 
! class="unsortable" | Immunities 
! class="unsortable" | Loot
|-
{{#cargo_query: tables=Bestiary
| fields=image, name, level, endurance, damage_lh, might, deflection, dr_base, ability, immunity, loot
| format=template 
| template=Displayformattest
| where=family='spore'
| order by=level, name
| named args = yes
}}
|}

Template, where I had to swap underscore for space

(Naturally this is WIP, and I need some colspan and other formatting)

<includeonly>
|-
|{{{image|}}}
|{{{name|}}}
|{{{level|}}}
|{{{endurance|}}}
|{{{damage lh|}}}
|{{{might|}}}
|{{{deflection|}}}
|{{{dr base|}}}
|{{{ability|}}}
|{{{immunity|}}}
|{{{loot|}}}
</includeonly>

The pages in question is this for output, and this for the template. In the SpecialPage CargoTable, the underscores are gone too. Pangaearocks (talk) 04:20, 6 September 2017 (UTC)

Are you possibly confusing between template fields and Cargo fields? They tend to match each other and have the same names, but their names are distinct from one another - Cargo's require underscores but are displayed with spaces, while template field names can have either. Or am I misunderstanding the question? Yaron Koren (talk) 13:26, 6 September 2017 (UTC)

Don't think so. The only place I am now using {{{dr base|}}} or dr base is in the template that deals with putting data from a cargo_query into a sortable table. Just to make sure, I searched for both terms in mainspace and template with Replace Text. Hits only for that page (plus its user testpage variant). Tried to revert back to using variables with underscore, and then the data is removed. For some reason I must use spaces instead of underscores, otherwise I can't locate the data in the output format. See two pictures for examples. First is with space (as in the rolled out versions), and second is with underscore (as the variables are called everywhere else). With space and With underscore Pangaearocks (talk) 06:32, 7 September 2017 (UTC)

Ah, now I get it. Yes, that seems like a bug. Yaron Koren (talk) 16:37, 7 September 2017 (UTC)

Error on DiscourseDB Bar chart page

Hi, for your information, I got an error when checking out the Bar chart example on DiscourseDB, referenced from https://www.mediawiki.org/wiki/Extension:Cargo/Display_formats.

Error: invalid table alias "Items".

Pangaearocks (talk) 19:41, 24 September 2017 (UTC)

Thanks for letting me know about that! That was an old query, which probably hasn't worked for a while. I fixed it. Yaron Koren (talk) 13:36, 25 September 2017 (UTC)

Creating a table that mixes data and Cargo DB data

I use Cargo to pull information about games from my database and display it in a table (one row per game). For example, see [3], where I specify 6 games that I want to be displayed in a table. I do this by using the "template" format option of cargo_query, and I allowed the "IN" keyword for Cargo's SQL queries.

Is it possible to add another static column to that table? That is, a column that contains data specified on-the-fly rather than pulled from the database? For example, I would like to add a "Price" column to that table, but I do not want the price to be stored in the database, but instead I want to specify it when I create the table (since I might create multiple different tables that pull information about the same games, but have different prices due to things like sales). Is there any way to do this?

Thanks in advance! NathanielJ (talk) 14:23, 28 September 2017 (UTC)

I think this used to be possible in #cargo_query but is no longer possible, due to a more restrictive approach to #cargo_query. It might be worth bringing that capability back - you're the first person that has asked about this sort of thing in a while. For now, the easiest approach might be storing that data via Cargo, but having the template only store the value and not display it, if the issue is that you don't want it displayed outside of the query. (What is the issue, out of curiosity?) Yaron Koren (talk) 15:41, 28 September 2017 (UTC)

Thanks for the info. The reason I don't want to store the data in the DB is that I will likely create multiple tables that might display different prices for the same game (e.g., I will create a news post on the wiki whenever games go on sale). I want the table to forever display that particular sale price, not the most up-to-date price associated with the game. NathanielJ (talk) 15:46, 28 September 2017 (UTC)

Ah, that makes sense. And I just realized that this is still doable - forget what I said before. It's a hack, but: add 'IF' to $wgCargoAllowedSQLFunctions in LocalSettings.php. Then, in the query, add to the "fields=" parameter a value like the following:

IF (_pageName = 'Chess', '$5', IF (_pageName = 'Checkers', '$4', IF (_pageName= 'Backgammon', '$7', 'Unknown' ) ) ) = Price

Please let me know if that worked for you. I should add something to the documentation about this... Yaron Koren (talk) 17:25, 28 September 2017 (UTC)

This works perfectly, thank you! One final follow-up question: is it possible to use CASE instead of IF to simplify this a bit? When I try to use CASE I get errors like "Error: no field named "" found for any of the specified database tables.", so I'm guessing that Cargo does some sanitization that makes it impossible. NathanielJ (talk) 19:13, 28 September 2017 (UTC)

Unfortunately, no - I tried that too. Cargo doesn't deal well with "CASE" because it's an operator and not a function (it isn't followed by parentheses). This could be fixed in Cargo, but it doesn't work at the moment. Yaron Koren (talk) 20:03, 28 September 2017 (UTC)

Tables not populating with data

I like Cargo a lot, but one frustrating aspect is that I'm having problems with tables not populating data, either when re-creating the table, or when making new pages. A while ago I re-created a table because it was given a new name. After 2-3 days it still hadn't re-populated all rows, so I tried to re-create it again. Now it's been another 2 days without the last 40-ish rows re-populating. On another table/template, all rows have disappeared. I inserted another field in the template, and edited and added pages transcluding it. Now all rows are gone. Do I need to re-create this table as well? Here is an overview of the tables. The problems are Bestiary_poe1 and Upgrade_poe1. Bestiary_poe1 should be 149 rows, and Upgrade_poe1 25 rows.

Isn't there some other way for Cargo to (re)populate tables? It seems to be a common problem with Cargo, from personal experience, others I've talked to, and looking through the archive of questions here. It's absolutely essential that this part works for sites using or migrating to Cargo, as otherwise pages will not look right, with missing data and error messages. Pangaearocks (talk) 15:41, 30 September 2017 (UTC)

Can't tell if this is related or a coincidence, but a staff member removed the transclusion of /doc pages, and suggested Cargo maybe didn't understand the structure and got confused, causing tables not to populate properly. At least currently, Upgrade_poe1 is up to the correct 25 rows. Bestiary_poe1 is currently going amok, tripling and quadrupling data, but I hope it cools down soon and reverts to the correct amount of rows, which should be 149.

Since we're on the 1.3.1 version, can the number in table names and/or fields cause this or other problems? I saw that was an issue which was fixed in 1.4 (hopefully we get it soon). Pangaearocks (talk) 23:46, 30 September 2017 (UTC)

Sorry about all the problems. My understanding is that you don't have command-line access - if you did, this would be a lot simpler, because then you could call cargoRecreateData.php, which works faster and more reliably. I think the main issue here is that jobs on your wiki are being run very slowly (probably due to a big backlog), which causes problems in two ways: it takes a long time for the tables to fill up, and it has caused you, out of frustration, to call a recreate on at least some of the tables more than once. Unfortunately, calling recreate a second time doesn't delete the remaining jobs from the first recreate, which leads to duplication of rows when all the jobs do finally get called. So, barring any change in the command-line circumstances, my advice would be to wait until the recreate jobs seem to have all been completed, then call "recreate data" on whatever tables are now too big, then be very patient and let the process just run.

The issue with numbers in field and table names was just for querying, so it wouldn't affect this - although of course it would affect querying of these tables. I don't fully understand the transclusion thing, but I'm guessing that's not an issue here either. Yaron Koren (talk) 01:28, 1 October 2017 (UTC)

Correct, we don't have command line access. However, I noticed yesterday that we do have some type of API access. I tried to run cargorecreatedata via that, but I suppose it's not the same thing, and maybe just another way to run Recreate data from e.g. the CargoTables overview page? This is what it looks like (a picture since you probably won't see what I see via the link). Not much happened when I ran it tbh, although the table did get cleared out when I ran cargocreatetables. That resulted in 0 rows though, and it's at 52 now, some 8-10 hours later.

What does appear to work is to manually save the problem pages. That's okay for small tables with few rows, but for the bigger ones with 100+ rows, that's not really feasible, especially in the long run. I did that on the duplicate rows for a couple of small tables, and that fixed it, so the table is correct atm. Also did that on a few pages that were missing from the big Bestiary_poe1 table. The re-saved page then instantly showed up in the cargoTable. Pangaearocks (talk) 16:12, 1 October 2017 (UTC)

Yes, calling that API action is the same as pressing the button in the web interface. And yes, resaving a page will get correct data for that page, though not for anything else. I would suggest talking to the administrators of that wiki about getting jobs to run faster - maybe by increasing the job run rate, or even calling runJobs.php manually. Yaron Koren (talk) 20:31, 1 October 2017 (UTC)

"Unfortunately, calling recreate a second time doesn't delete the remaining jobs from the first recreate, which leads to duplication of rows when all the jobs do finally get called." -- I had never thought of this as the cause of the duplicate rows problem. I had that sometimes and it seemed to be after trying to recreate tables. Thanks for explaining. Jonathan3 (talk) 00:46, 11 October 2017 (UTC)