Extension talk:Cargo/Archive August to September 2018

Error: 1069 Too many keys specified; max 64 keys allowed

Recreating a CargoTable via the cargoRecreateTable.php script gave me the following error: Error: 1069 Too many keys specified; max 64 keys allowed. The table contains 61 fields.

The query that threw the error is: Query: CREATE INDEX progress_75_date__precision_webmo_indicator ON `cargo__webmo_indicator` (`progress_75_date__precision`)

Cargo always seems to create indexes for all table columns and also a sort of __precision index (I don't understand why) which eventually leads to a number of keys larger than 64 for this specific table.

A solution could be to disable indexing (apart from applying a leaner database design). Is there a way to disable indexing in Cargo? Would that make sense?

Indexing of all fields was add just a month ago - I'm glad you found this issue now, before the next version is released. I didn't know about that keys limit that MySQL has. I just checked in a fix for this, so if you get the latest code and recreate the table again, hopefully it will work. Yaron Koren (talk) 14:54, 7 August 2018 (UTC)

Thanks for the quick reply and the fix! We'll implement it this week and i'll let you know. I did not know of the key limit either. Apparently the key limit can only be changed by recompiling the code. I'd be interested in the reason for enabling indexing on all fields? Did you notice performance improvements? --Schtom (talk) 09:21, 8 August 2018 (UTC)

I misspoke before - not all fields are indexed; just fields that have a type other than "Text", "Wikitext" and "Searchtext". (And for date fields, there's an extra field, for the precision, that is also indexed.) The idea behind this is to get queries to run faster in general. For any specific table in any specific wiki, there may be only one or two fields where an index will make anything run faster - depending on the size of the table, etc. I could have made indexing a setting in #cargo_declare, to let admins decide which fields should get indexed and which should not. However, I figured it would be easier for admins, and more foolproof, to just index every field that could conceivably be used for sorting or joins. That's especially true given that I think most people running Cargo will not be DB experts. There's a tradeoff there in terms of slower writes to the DB, but I think it's worth it - on a wiki, read speed is much more important than write speed. Still, this is a very new feature, and I'm not sure it's the correct approach - I'd be glad to hear your thoughts on the matter. Yaron Koren (talk) 14:38, 8 August 2018 (UTC)

Thanks for the explanation and sorry for the late reply. We also did not implement your fix (yet), as our client decided to change his database design. Personally, I do think your approach and reasoning makes a lot of sense. However, I cannot assess the actual performance improvements. I wanted to let you know about another issue that occurred with the same client. He reported a database error when creating a new table. We then found that the identifier limit is restricted to 64 chars (Error: 1059). One of the identifiers was throwing the error was 'deadline_minigrid_people__precision_webmo_indicator_minigrid_people' (67 chars). The composed index keys may become an issue as well. It seems there could be a trade-off between usability/readability (users are forced to use shorter names) and performance. So far we have not rolled out our cargo-driven wiki and are merely beta-testing with this one client. These two issues (64 keys, 64 chars) were the only ones reported to us.--Schtom (talk) 13:53, 21 August 2018 (UTC)

Those are some pretty long table and field names! Anyway, thanks for letting me know about the problem, and sorry about that. I just checked in what I think is a fix. Yaron Koren (talk) 03:45, 22 August 2018 (UTC)

Kindly note that we do not consider these two issues as major problems. We find your extension incredibly useful. I agree that both the field and table names are pretty verbose. Still, I assume the identifier error was thrown by one of the index keys that is composed of both table and field name + underscores + the 'precision' label that you add. As a suggestion: You could maybe shorten the latter to leave a little more 'explanatory space' for the user e.g., 'pr' instead of 'precision'. --Schtom (talk) 10:38, 23 August 2018 (UTC)

That's great to hear! Yes, reducing the size of "precision" could make sense. I'll see how the current solution works out. Yaron Koren (talk) 15:27, 23 August 2018 (UTC)

Ad block (uBlock Origin) blocks Cargo script for displaying bar charts

On this page I noticed if I have uBlock Origin enabled the bar charts do not load. Disabled uBlock and the bar charts load normally.

Chrome dev console reveals this error:

Uncaught TypeError: Cannot read property 'disabled' of undefined
   at String.<anonymous> (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:458)
   at Array.d3_transitionPrototype.filter (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:221)
   at SVGSVGElement.<anonymous> (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:458)
   at load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24
   at d3_selection_each (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24)
   at Array.d3_selectionPrototype.each (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24)
   at Array.chart (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:455)
   at Array.d3_selectionPrototype.call (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:24)
   at Object.generate (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:562)
   at renderLoop (load.php?debug=false&lang=en&modules=ext.cargo.nvd3&skin=hydradark&version=0a89kh4:246)

Please advise. --pcj (talk) 19:47, 10 August 2018 (UTC)

Never mind, this was a Gamepedia-specific issue and has been resolved. --pcj (talk) 02:18, 2 September 2018 (UTC)

Good news! I had no idea what was causing the problem. Yaron Koren (talk) 13:03, 2 September 2018 (UTC)

SMW subobject transitioning to Cargo, table display issues

I'm trying to move a wiki from SMW to Cargo and one of the things we use is subobjects. E.g. A character may need two types of materials to ascend to the next stage -- 4 feathers and 5 fluid. Ideally this would be stored as a list with entries that have properties. i.e. Feathers (4);Fluid (5).

How do I do this in Cargo, or does anyone have a subobject-like store they could show an example of, please? Thanks!

Toriayl (talk) 14:21, 14 August 2018 (UTC)

One of the nice things about Cargo is that all data is stored in the same way, via #cargo_store - there's no distinction between regular storage and "subobjects". In other words, assuming you store all data via templates, it doesn't matter whether there's one template call on the page, or multiple calls. Does that make sense? Yaron Koren (talk) 17:27, 14 August 2018 (UTC)

Thanks for the reply, but sorry, still not really getting it. The point of sub objects is to link data to specific parameters entered. E. G. An address would have its post code and road number linked to it. Are you saying that my storage needs to be multi row within each page to accomplish similar functionality? I currently have a single template that loops through items and amounts entered and stores each as subobjects. I can't really use multiple templates instead, but maybe I need to figure out how to store them on different rows within the same template.. Was hoping there would be a single-row solution.

Alternatively, am I meant to store directly/is it possible to do so into a helper table for a field containing a list of values? Toriayl (talk) 22:25, 14 August 2018 (UTC)

Oh - I didn't realize you have a single big template that's storing an entire "table" of values. You can still use Cargo for this - you would just need to create a new template that holds one row of that table, then have the big template call that template in each place it previously called #subobject. Does that make sense? Yaron Koren (talk) 01:42, 15 August 2018 (UTC)

Thank you, I was hoping to have the info in one table but it sounds like another table would be a better idea for this. And rows would look like:

Artemis - Feather - 1
Artemis - Feather - 5
Artemis - Fluid - 5

Hopefully that's on the right track but I'll worry about it later. I have more questions for now! (Sorry. XD) Not really subobject-related now, though.

1. Is there a way to pull just the first result of a joined table? i.e. If Artemis matches with two entries on another table, and I want just something from the first entry matched. This one is not 100% needed; I'll probably manually input the values needed into each page (200 of them...), but I think I've read something about SQL being able to do this (though not very cleanly) so thought I might as well ask in case there's something for Cargo I've missed.
2. I am having some trouble generating a table, particularly with the values in a list on it. This is the table: https://grandorder.wiki/Special:CargoTables/CEs Related template: https://grandorder.wiki/index.php?title=Template:CEInfoBox&action=edit Test page with issues: https://grandorder.wiki/User:Toriayl/sandwich2

• The Effects parameter is stored in Effects=List (;) of String, which appears to work fine on the Special:CargoTables table-browsing link above.
• However, when I try to put it in a dynamic table query, I get a database error if I put it as the last item displayed.
• The query works if I put it as the first or second item displayed, but lists with three items have the second item within pre tags. Something to do with the dot delimiter..?
• Using template as an alternative, effects work but html tags (br, div style etc) are displayed.

Help would be really appreciated, thanks! Toriayl (talk) 14:20, 29 August 2018 (UTC)

Alright - 1) there may be a way to do with some hack, maybe using "group by=", depending on the exact structure of the data - I'm not sure. 2) I'm not entirely sure what's causing the strange behavior with the Effects field, but I think it's at least partly due to the somewhat unusual way in which that value is put together. The "cleaner" way to do it would be again with a multiple-instance template holding the effects and "LBs" - so that each effect is directly connected to its LB value, and so that you don't need to have an artificial limit on the number of effects. But barring that change, you may be able to get around this problem by replacing "Effects" in the query with "Effects__full=Effects". Yaron Koren (talk) 15:27, 29 August 2018 (UTC)

Thanks, but I don't get how it's unusual...isn't it a fairly normal list? Effects__full=Effects got around the database error, but it's still showing the 'second item pre' issue. Also pardon me if I'm wrong, but that's the same fix you were suggesting for subobjects, isn't it? i.e. Create another table with the effects, then join it with the main table.

But won't this cause the same problem as 1)? I want to only display one line with all the effects for an item, I don't want three lines for three effects, with all of the other info duplicated......

Toriayl (talk) 13:01, 30 August 2018 (UTC)

Yes, that's the same fix I suggested. In both cases, you have a table of data stored on the page, so it should be stored in a tabular fashion. When you display the data, you can display it all on one line, even if it represents three different "rows". What do you mean by "all of the other info duplicated"? Yaron Koren (talk) 19:18, 30 August 2018 (UTC)

How do I display it all on one line?! Have a look on my sandbox again, the first section. This is the only way I know how to query/display the table, and it shows multiple lines, one for each match in the join...and results in a duplication of the other information (rarity, ID, etc).

Toriayl (talk) 11:41, 1 September 2018 (UTC)

If you add "|format=list" to the query, it will display all results on one line. You can also do that with the "template" format, which gives you complete control over the display, although with that you have to create a template. The duplication thing is a problem, though it's a problem in the storage, not the querying. I wish I knew how to prevent duplicates in all cases. In any case, if you do a recreate data on the "ServantsNP" table again, it will probably fix the problem in that table. Yaron Koren (talk) 02:27, 2 September 2018 (UTC)

Thanks for trying to help, but it feels a little like we're on different wavelengths here. ^^" I have one page that has two rows associated with it. I ran a recreate data on the ServantsNP table and it gives me the same thing. E.g. Alexander B+ NP, Alexander B++ NP. This is correct. Alexander is linked to two different types of NP. But I want it to display, instead of

Alexander	B
Alexander	B+

on one line:

Alexander

B, B+

It doesn't look like format=list or template can do this? Toriayl (talk) 15:41, 5 September 2018 (UTC)

Displaying a table on a page for searching by values

Maybe someone already asked such a question.

Is it possible to create a table on a page with search? or create such a table similar to Drilldown. imgur - (Oleksiy) 10:47, 22 August 2018 (UTC)

Do you know about the "dynamic table" format? Yaron Koren (talk) 16:09, 22 August 2018 (UTC)

Cargo not updating tables, nothing happening.

Hi, I am having an issue where cargo is not updating tables. Only some pages are being successfully stored while others aren't, despite there being no real difference. Existing pages also do not update despite the job queue being empty. Steps taken:

• Recreated table using ?action=recreatedata - nothing happens but jobs are added, jobs will never progress (even with high run rate.) Also tried manually recreating data.
• Manually ran jobs through the server, created only half of the data then stopped. No jobs left.
• Recreated data on tables that had no population whatsoever - no jobs created.
• Manually ran jobs through server, nothing happens.
• If existing data on the table is updated, it does not update or create a job.

Semantic Mediawiki worked flawlessly beforehand, no changes were made between the two except for an update from MW 1.29 to 1.31 , any idea to what the issue may be? Steps were taken from the FAQ and common problems as well, none seem to be the issue. -- 2607:FEA8:E2E0:602:CCB2:DD2C:F53E:4C48 08:17, 24 August 2018 (UTC)

I didn't understand all of this. What do you mean by "manually recreating data", and what happened when you tried it? What do you mean by tables with "no population whatsoever" - empty tables? If so, what should have happened that didn't happen? What do you mean by "existing data" being updated - a regular page save? And when you wrote "Semantic Mediawiki", did you mean Cargo? If not, I don't see the connection. Yaron Koren (talk) 13:22, 24 August 2018 (UTC)

Sorry, let me be more clear, I am not 100% on the terminology as of yet:

1. We used SMW before transferring over to Cargo, just updated templates using the migration guide. Only the first time we recreated data using the web browser then ran runJobs.php did it partially work. It did not grab every page's data and stopped midway. (SMW is completely disabled at the moment.)
2. "Manually created data" by running runJobs.php / php cargoRecreateData.php
3. The tables specifically aren't being updated, despite it should be picking up the changes, the jobs aren't being created/aren't functioning correctly.
4. I have a local copy of Cargo and confirmed the templates used work and will successfully create the table and fill them with data, its only on the server does it not, despite there being no errors.

I hope this helps, I am completely stumped. Is there any further debugging I can do? -- 2607:FEA8:E2E0:602:4D63:99F6:223C:D164 04:45, 26 August 2018 (UTC)

I just finally tried running Cargo on MW 1.32.0-alpha, and it seems to be working fine for me, including recreating data. So I don't know what's going wrong. Just so I understand - were you using Cargo with MW 1.29? And what version of Cargo are you running? Also, this isn't a public wiki by any chance, is it? Yaron Koren (talk) 03:44, 29 August 2018 (UTC)

Heres the link to the wiki, all the information should be on there. As an example, the machine table only has 4 entries, when 20-ish pages actually use the template and should be storing data. --2607:FEA8:E2E0:602:68B1:ACDA:6EA3:A27 22:49, 29 August 2018 (UTC)

Alright. Do you see any error messages when you run cargoRecreateData.php? Yaron Koren (talk) 18:15, 31 August 2018 (UTC)

No error messages, says its "done" but doesn't actually do anything. -- 2607:FEA8:E2E0:602:B01E:BB64:DD4F:C263 20:51, 1 September 2018 (UTC)

Additionally : I can get this to work on a local copy but not the actual server. That being said I have no personal access to the server, so it's difficult to diagnose. -- 2607:FEA8:E2E0:602:D9EB:B7E9:C8A0:6D6D 01:10, 3 September 2018 (UTC)

Is there any difference in versions between the local copy and the actual server? Also, if you don't have access to the server, how were you able to run cargoRecreateData.php? Yaron Koren (talk) 02:48, 3 September 2018 (UTC)

It's a weird administration system, I am able to ask the server admin to debug some stuff for me, but they are very slow at doing so and won't give me access - so the only information I get is basic. It makes this process much more difficult and frustrating. There is no difference in the versioning. -- 2607:FEA8:E2E0:602:1569:7DD9:88C1:6AD0 20:15, 3 September 2018 (UTC)

Update: I found out SMW was improperly uninstalled, after properly having it removed, cargorecreatedata.php started working and successfully filled tables. However, tables will still get stuck trying to populate via the web interface, and they will not update automatically if a page is edited. -- 2607:FEA8:E2E0:602:F5FA:7A1F:1CDF:59B3 19:03, 6 September 2018 (UTC)

Update 2: Debugging further found that jobs aren't being created by cargo to do the work, despite saying Note: One or more of these tables are currently being populated, via the job queue. The showjobs.php shows 0 jobs and none failed / abandoned. -- 2607:FEA8:E2E0:602:6134:8886:DF34:6CF7 21:23, 8 September 2018 (UTC)

This sounds a lot like the issues we have on Gamepedia. Does it recreate the data as expected when you blank edit the pages that store? --RheingoldRiver (talk) 22:25, 4 September 2018 (UTC)

No, it doesn't. -- 2607:FEA8:E2E0:602:A012:9E7D:B5D3:26CE 17:55, 5 September 2018 (UTC)

It's great that you were able to get cargoRecreateData.php to work, and that you diagnosed the issue with the web interface (it's not creating jobs). I don't know why jobs aren't being created, though. Do you know if other processes that create jobs are working on your wiki? For example, modifying a template should cause a job to get created for each page that calls that template, to refresh its contents - does that happen? Yaron Koren (talk) 01:42, 9 September 2018 (UTC)

Yes, other jobs are being created and work correctly, cargo does not. Cargo says it has created jobs at this page, but showjobs.php shows 0 jobs. Non-cargo jobs are also being processed correctly. -- 2607:FEA8:E2E0:602:C8E9:DA9F:601F:C5F 20:31, 10 September 2018 (UTC)

Hello, I'm having the same issue on https://dicoado.org/dico/Sp%C3%A9cial:CargoTables I was using SMW until now. First time I rewrote the template to use Cargo it worked fine but my variables weren't optimaly chosen. I deleted the tables and started fresh again. As you can see, when I used cargoRecreateData.php, only 16 articles were included in the table (it should be over 2k). Although it says other jobs are being created, cargo does not. DSwissK (talk) 05:36, 13 September 2018 (UTC)

At the moment, it looks like you have an even bigger problem - the cargo_tables DB table was deleted, but it shouldn't have been. You should call MediaWiki's update.php script again to restore it. Yaron Koren (talk) 15:09, 13 September 2018 (UTC)

Thanks for the fast answer. Yes, I was working on it. :) I found the issue. Somehow, I had only 20ish pages linked to my template when calling the special page so I used refreshLinks.php and now it's working like a charm ! :D Thanks ! DSwissK (talk) 19:35, 13 September 2018 (UTC)

Great! Yaron Koren (talk) 18:43, 14 September 2018 (UTC)

Cargo not updating tables

Hello Yaron, I have the same issue with mw1.31. : "Cargo not updating tables, nothing happening". The table is created with a form and a template (using #cargo_declare in the template) the data is store in the mw page (using #cargo_store in the template) when I save the form but the cargo table is empty. I used a separate cargo database. The only way I have found to populate the cargo table is to run "php cargoRecreateData.php" in the Cargo's /maintenance directory. (Jacques Masbou)

What version of Cargo are you running? Is this a public wiki? Did it work better with an earlier version of MW? What database are you running? Does everything work fine if you call cargoRecreateData.php? Yaron Koren (talk) 16:48, 29 August 2018 (UTC)

Cargo version 1.7 php 7.22 mw1.31 It is not a public wiki mysql 5.7.19 when I call cargoRecreateData.php sometimes it works sometime it does not work With cargo 1.5, php 5.6.31 mysql 5.7.18 and mw 1.29 it works fine

Update : with mw 1.29.2, php 7.1.21 mysql 5.7.19 cargo 1.7 it works fine

It's great that you isolated that the problem is due to the MediaWiki version, not the Cargo version. Still I don't know what's causing the problem - I'm running Cargo on MW 1.31 myself, and it seems to run fine. Do you have more than one Cargo table on your wiki? If so, is the problem happening in all of them? Yaron Koren (talk) 03:00, 2 September 2018 (UTC)

We have a similar issue, and a possible related issue which may give some clues. Cargo 1.7, PHP 7.0.31 (fpm-fcgi), Mediawiki 1.31.0.

When updating a template no job appears to be created to update the table data. When we run the maintenence/recreateData.php script it works as expected and creates the tables. However, when we run the "recreate data" command through the web ui on a table, it creates the table just fine but we are unable to successfully delete the tables, or switch to the new tables.

Recently we had an issue with the AWS extension which turned out to be due to the fact that our MediaWiki is configured to use a prefix on database tables. It is possible that this is a similar issue, since database prefixing is not standard wiki setup and usually not noted when it is? Ian T Small (Ian T Small|talk) 18:00, 18 September 2018 (UTC)

Just to be clear - calling cargoRecreateData.php and calling "Recreate data" from the wiki both work, but the latter leads to a situation where you can't delete the table, or switch in its replacement table? Yaron Koren (talk) 19:48, 18 September 2018 (UTC)

Error when creating a new data table

Hi,

I got this error when trying to create a new data table (api.php?action=cargorecreatetables&template=Artiste):

{
    "error": {
        "code": "internal_api_error_DBUnexpectedError",
        "info": "[6377a1755353fc18463173ed] Exception caught: Wikimedia\\Rdbms\\Database::close: mass commit/rollback of peer transaction required (DBO_TRX set).",
        "*": "Wikimedia\\Rdbms\\DBUnexpectedError at /var/www/wiki/w/includes/libs/rdbms/database/Database.php(916)\n#0 /var/www/wiki/w/extensions/Cargo/includes/CargoUtils.php(818): Wikimedia\\Rdbms\\Database->close()\n#1 /var/www/wiki/w/extensions/Cargo/includes/CargoUtils.php(534): CargoUtils::createCargoTableOrTables(Wikimedia\\Rdbms\\DatabasePostgres, Wikimedia\\Rdbms\\DatabasePostgres, string, CargoTableSchema, string, integer)\n#2 /var/www/wiki/w/extensions/Cargo/api/CargoRecreateTablesAPI.php(31): CargoUtils::recreateDBTablesForTemplate(integer, boolean)\n#3 /var/www/wiki/w/includes/api/ApiMain.php(1579): CargoRecreateTablesAPI->execute()\n#4 /var/www/wiki/w/includes/api/ApiMain.php(535): ApiMain->executeAction()\n#5 /var/www/wiki/w/includes/api/ApiMain.php(506): ApiMain->executeActionWithErrorHandling()\n#6 /var/www/wiki/w/api.php(94): ApiMain->execute()\n#7 {main}"
    }
}

Looking at the SQL logs (I am using PostgreSQL), I can see the correct CREATE TABLE statements, but no COMMIT:

2018-08-25 19:17:49 CEST [14818-22] wiki@wiki LOG:  statement: CREATE /* Wikimedia\Rdbms\Database::query  */ TABLE "cargo__Artistes__Genres" ( "_rowID" Int, "_value" Varchar(300) )
2018-08-25 19:17:49 CEST [14818-23] wiki@wiki LOG:  statement: CREATE /* Wikimedia\Rdbms\Database::query  */ INDEX "row_id_Artistes__Genres" ON "cargo__Artistes__Genres" ("_rowID")
2018-08-25 19:17:49 CEST [14818-24] wiki@wiki LOG:  statement: ROLLBACK /* Wikimedia\Rdbms\Database::close  */ 
2018-08-25 19:17:49 CEST [14817-31] wiki@wiki LOG:  statement: ROLLBACK /* MWExceptionHandler::rollbackMasterChangesAndLog  */ 
2018-08-25 19:17:49 CEST [14817-32] wiki@wiki LOG:  statement: BEGIN /* Wikimedia\Rdbms\Database::query (LCStoreDB::get)  */ 

Any help is welcome :)

--Erdnaxeli (talk) 17:25, 25 August 2018 (UTC)

Ok, I tried with the git master branch (instead of the latest zip) and it seems the bug was fixed. --Erdnaxeli (talk) 15:37, 31 August 2018 (UTC)

That's great to hear - I was planning to set up Cargo on Postgres to try to debug the issue, but I guess now I don't need to. Yaron Koren (talk) 18:14, 31 August 2018 (UTC)

File field : edit redlink instead of File:myFile

I am using

• Special:CreateClass

To obtain Table structure:

• bebe - String
• myfile1 - File
• bibi - String
• myfile2 - File

then

• Form:MyFileB1

has uploadable for File fields.

Page creation and file uploading is working, But my page shows a redlink for File field

• index.php?title=Blue16Test1A.jpg&action=edit&redlink=1

I think it should it be linking to

• File:Blue16Test1A.jpg

Would you help me to link to uploaded file ?

Thanks in advance. --Wpe (talk) 15:15, 26 August 2018 (UTC)

Yes, the template should have the string "File:" before the template field holding the filename, so it should look like "File:{{{MyFile|}}}". Sorry, I realized that Special:CreateClass and Special:CreateTemplate don't put in that necessary prefix when creating the template; I just checked in a fix to Page Forms, so hopefully they will now. Yaron Koren (talk) 03:22, 27 August 2018 (UTC)

Missing _pageData

How to solve this problem?

[9f30956ef6e5b7f44122218a] /%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:CargoTables/_pageData MWException from line 196 of /home/hydra/public_html/extensions/Cargo/includes/CargoUtils.php: Error: Table _pageData not found.

Backtrace:

#0 /home/hydra/public_html/extensions/Cargo/specials/CargoTables.php(86): CargoUtils::getTableSchemas(array)
#1 /home/hydra/public_html/includes/specialpage/SpecialPage.php(522): CargoTables->execute(string)
#2 /home/hydra/public_html/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run(string)
#3 /home/hydra/public_html/includes/MediaWiki.php(287): SpecialPageFactory::executePath(Title, RequestContext)
#4 /home/hydra/public_html/includes/MediaWiki.php(862): MediaWiki->performRequest()
#5 /home/hydra/public_html/includes/MediaWiki.php(523): MediaWiki->main()
#6 /home/hydra/public_html/index.php(43): MediaWiki->run()
#7 {main}

--Ruba159753 (talk) 13:54, 4 September 2018 (UTC)

Right, the _pageData table doesn't get created automatically - instead you have to create it by defining its columns, then calling a script. See here. Yaron Koren (talk) 19:19, 4 September 2018 (UTC)

Declaration Parameter to Index a Field

Would it be possible to add a flag in declaration of a field that makes the column indexed (but not unique)? --RheingoldRiver (talk) 18:49, 5 September 2018 (UTC)

Actually, in the latest Cargo code, every field now gets indexed - except for fields of type Text, Wikitext and Searchtext, and fields after #50 or so in tables with a large number of fields. Yaron Koren (talk) 23:16, 5 September 2018 (UTC)

Ah ok, we are still not on latest version until after MediaWiki upgrade. Good to know, thanks. --RheingoldRiver (talk) 00:57, 6 September 2018 (UTC)

Error: 1054 Unknown column '_pageData_alias._fullText'

Hi, I've been getting this error on a full text search. On investigation it looks like your more complex demonstrations of the MATCHES are suffering from the same issue. see error reports on http://discoursedb.org/wiki/Search_demo

Sorry about that - that's due to some recent changes that weren't fully tested. I just checked in what I think is a fix for this problem. Yaron Koren (talk) 14:01, 18 September 2018 (UTC)

Thanks for the quick fix. James Driscoll

Proposal for hook 'CargoBeforeStoreAllData'

Can I suggest the addition of a hook in CargoPageData.php ?

a simple call to

       Hooks::run( 'CargoBeforeStoreAllData', array( &$title, &$pageDataValues, &$tableSchemas['_pageData'] ) );

before storing the data

	CargoStore::storeAllData( $title, '_pageData', $pageDataValues, $tableSchemas['_pageData'] );

this would enable other extensions to modify the fulltext data (and other params) before insertion, e.g. to sanitise text or add additional data to to a full text search

mediwiki wouldn't let me upload a patch file but here are the contents...

--- CargoPageData.php.original	2018-08-26 21:32:15.000000000 +0100
+++ CargoPageData.php	2018-09-19 16:03:24.115780698 +0100
@@ -137,6 +137,7 @@
 			$pageDataValues['_isRedirect'] = ( $title->isRedirect() ? 1 : 0 );
 		}
 
+		Hooks::run( 'CargoBeforeStoreAllData', array( &$title, &$pageDataValues, &$tableSchemas['_pageData'] ) );
 		CargoStore::storeAllData( $title, '_pageData', $pageDataValues, $tableSchemas['_pageData'] );
 	}
 

Thanks. James Driscoll

I'm always happy to add hooks into the code, and this one seems useful. One question, though: would it be better to put the hook inside the storeAllData() function (right at the beginning of it)? storeAllData() is called in three places: for regular data from #cargo_store, for "page data", and for "file data". The table name could be passed in to the hook, letting the outside code decide whether to modify the data for any one table. What do you think? Yaron Koren (talk) 16:14, 21 September 2018 (UTC)

that sounds like a better plan Yaron. Thanks ,James Driscoll

Okay - actually, now I'm not sure about this placement either. Are you already using such a hook locally? Because, looking through the code, it seems to make more sense to place it after the values have already been set by the code, but before they're stored in the database - maybe around line 331 of CargoStore.php, or halfway through the function. Otherwise, there's a good chance that the values set via the hook will get overridden, unless there are some serious hacks put in. On the other hand, if it's already working fine for you, then that's a different story. Yaron Koren (talk) 02:29, 24 September 2018 (UTC)

Hi Yaron, yes it's already working fine locally for me but I'm only modifying the '_fullText' field which I don't think you're changing? It makes sense for it to be as late as possible so values aren't overridden. James Driscoll

Ah, that explains it. Okay, I just added a hook, now called "CargoBeforeStoreData", which has the following structure:

Hooks::run( 'CargoBeforeStoreData', array( $title, $tableName, &$tableSchema, &$tableFieldValues ) );

It would be great if you could get the latest Cargo code, then adapt your own code to this new structure, and see if it still works. Yaron Koren (talk) 16:43, 24 September 2018 (UTC)

Works great thanks. I hope it's useful to others JamesDriscoll (talk) 19:38, 25 September 2018 (UTC)

Good to hear! And it's quite possible. Yaron Koren (talk) 19:51, 25 September 2018 (UTC)

Parser state cleared while parsing. Did you call Parser::parse recursively? on Special pages

We have a special page in an extension that relies heavily on cargo queries and are hitting this problem. "Parser state cleared while parsing. Did you call Parser::parse recursively? on Special page"

as a hack, in CargoUtils line 494

		// This next clause should only be called for Cargo's special
		// pages, not for SF's Special:RunQuery. Don't know about other
		// special pages.
		} elseif ( ( $title != null && $title->isSpecialPage() && !$wgRequest->getCheck( 'wpRunQuery' ) ) ||
			// The 'pagevalues' action is also a Cargo special page.
			$wgRequest->getVal( 'action' ) == 'pagevalues' ) {
			$parserOptions = new ParserOptions();
			if ( method_exists( $parserOptions, 'setWrapOutputClass' ) ) {
				// Remove '<div class="mw-parser-output">' from around
				// the value, if it was parsed - this class, and method,
				// were both added in MW 1.30.
				$parserOptions->setWrapOutputClass( false );
			}
			$parserOutput = $parser->parse( $value, $title, $parserOptions, false );
			$value = $parserOutput->getText();
		}

we are checking if the page is specifically our extensions special page and calling

 $parserOutput = $parser->parse( $value, $title, $parserOptions, true, false );

rather than your call of

 $parserOutput = $parser->parse( $value, $title, $parserOptions, false );

Similarly, to avoid the same error, in CargoTemplateFormat line 73

we are checking if the page is our specialpage and just returning $text rather than calling CargoUtils::smartParse

	if ( $wgTitle != null && $wgTitle->isSpecialPage() ) {
			return CargoUtils::smartParse( $text, $this->mParser );
		} else {
			return $text;
		}

are these checks meant to be specifically for landing on the cargo special pages rather than anyone else's special page?

Interestingly, on line 481 of CargoUtils, there is the comment that might be suggesting this is the case...

// This next clause should only be called for Cargo's special
// pages, not for SF's Special:RunQuery. Don't know about other
// special pages.

I hope this makes sense! JamesDriscoll (talk) 09:43, 27 September 2018 (UTC)

Sorry, I don't understand. What changes did you make to the code, and did they fix the problem? Yaron Koren (talk) 21:56, 27 September 2018 (UTC)

here are the patches. Although they work for my particular case I'm not suggesting these as the fix. I'm wondering if you should be explicitly checking in the code if you are landing on a cargo special page before calling smartparse and $parser->parse( $value, $title, $parserOptions, false ); JamesDriscoll (talk) 06:56, 28 September 2018 (UTC)

--- CargoTemplateFormat.php.original	2018-09-24 16:58:23.000000000 +0100
+++ CargoTemplateFormat.php	2018-09-26 18:39:28.684640811 +0100
@@ -69,11 +69,13 @@
 			$text .= $this->displayRow( $templateName, $row, $fieldDescriptions, $namedArgs );
 		}
 		global $wgTitle;
-		if ( $wgTitle != null && $wgTitle->isSpecialPage() ) {
+ //hack to stop error on recursive call to Parse on the utilities search page
+		if ( $wgTitle != null&& $wgTitle->isSpecialPage() && $wgTitle->getText()!=='Utilities/Search'  ) {
 			return CargoUtils::smartParse( $text, $this->mParser );
 		} else {
 			return $text;
 		}
+		
 	}
 
 }

--- CargoUtils.php.original	2018-09-24 16:58:23.000000000 +0100
+++ CargoUtils.php	2018-09-26 18:29:07.542333280 +0100
@@ -491,7 +491,16 @@
 				// were both added in MW 1.30.
 				$parserOptions->setWrapOutputClass( false );
 			}
-			$parserOutput = $parser->parse( $value, $title, $parserOptions, false );
+
+	   //hack to stop error on recursive call to Parse on the  utilities search page
+	  	if ( $title != null && $title->isSpecialPage() && $title->getText()==='Utilities/Search'  ) {
+  			$parserOutput = $parser->parse( $value, $title, $parserOptions, true, false );
+    }
+    else
+    {
+  			$parserOutput = $parser->parse( $value, $title, $parserOptions, false );
+    }
+
 			$value = $parserOutput->getText();
 		} else {
 			$value = $parser->internalParse( $value );

Sorry for the long delay. What is Special:Utilities/Search, if I may ask - is it part of a released extension? Is its source code public? Yaron Koren (talk) 16:47, 9 October 2018 (UTC)

Old vs new layout in drill-down (and forms)

Hi Yaron,

I have noticed that you have nice looking search-boxes and buttons and other values-boxes in drill-down at Discourse DB. The same goes for forms created in Page Forms. In my drill-down (Cargo 2.0-alpha) I have the old looking grey buttons and the same in forms (Page Form 4.4-alpha). I understand this has to do with a setting, but where and how do I change this to the newer looking layout? Pretor~nowiki (talk) 11:12, 28 September 2018 (UTC)

No, it's not a setting - it may just have to do with your MediaWiki version. What version of MW are you using? Yaron Koren (talk) 16:16, 28 September 2018 (UTC)

I'm using MediaWiki v1.31.1 and the issue has been there since the website was established with v1.30.0. It's only on Cargo and Page Forms related pages that this issue occurs, the rest of the website has the up-to-date layout with blue buttons etc. Pretor~nowiki (talk) 17:35, 28 September 2018 (UTC)

Oh, okay. The issue might be your version of Cargo - there actually is newer code than version 2.0-alpha. I would try upgrading your Cargo to the latest code. Yaron Koren (talk) 20:04, 28 September 2018 (UTC)

Upgraded Cargo from git (2.0-rc1) and that fixed the problem in drill-down, thank you :) Did the same with Page Forms (4.4-rc1), but that did'nt help for the forms. Should I post the same question in the Page Form talk page? Pretor~nowiki (talk) 20:53, 28 September 2018 (UTC)

I don't remember any big change in the display of Page Forms forms... was there? Yaron Koren (talk) 22:24, 28 September 2018 (UTC)

You have probably not updated Page Forms to use the "new" MediaWiki layout with blue buttons etc. No big deal and thanks for the tip that fixed this in drill-down! Pretor~nowiki (talk) 09:43, 29 September 2018 (UTC)