Parsoid/Deployments/2014

Thursday, Dec 11, 2014 around 3pm PST Deployed d16dd2db

Disable caching for the v2 entry point (currently only used by RESTbase)
Don't allow quotes in generic_attribute_name
task T75909 Use localized main page name
task T50900 Add mw:Error + error info to data-mw for missing imgs

Wednesday, Dec 3, 2014 around 2pm PST Deployed 733986a6

task T75526: Fix crash while expanding templates on older wikis
task T73306: Pass the revid when expanding templates
task T75121: Infer extension name from typeOf if data-mw not present
task T76518: Fix failures from v2 endpoint
Fix config clobbering via v2 API

Wednesday, Nov 26, 2014 around 2pm PST Deployed 67e2596c

Subpage link fixes
Add wikidata API URL
Add wikispecies API URL

Monday, Nov 17, 2014 around 1pm PST Deployed 819b2cf4

bug 65870: Expose the list of languages from ParsoidConfig
bug 70867: Set xmlns namespace URI on elements that reference it
bug 50263: Fix a.args crasher
Fix /_rtform endpoint (this is not a published or stable API).
Add new language code mai.
bug 73396: Logging improvements.
bug 50121: Return 404 when page does not exist.

Thursday, Nov 13, 2014 around 1pm PST Deployed dabff010

Properly escape question marks in page titles in <head>
Use correct relative link prefix for some Category links
bug 70743: Point base href to wiki base.

Monday, Nov 10, 2014 around 1pm PST: Deployed b61475196

Enable cpu timeouts for sufficiently new versions of node
Document the expected behaviour for the timeouts
Tweaked serializer error messages

Thursday, Nov 6, 2014 around 2pm PST: Deployed d23d2be60

Update prfun to v1.0.2
More efficient promise use for timeouts
Add support for foundationwiki

Wednesday, Nov 5, 2014 around 1pm PST: Deployed 978623eb

bug 57252: Generate HTML5-compliant cite id/about attr values
bug 67657: Add normalized parameter names to templates
Caching related bug fix in some extension/template reuse scenarios
Logging: Log process events to LogStash
Logging: Only send warning and more severe events to LogStash (to reduce load on LogStash for now)
Logging: Downgrade some old error events to warnings

Monday, Nov 3, 2014

No deploys today. Cluster upgrade (ubuntu and node 0.8.x -> 0.10.x) being tested / monitored on a single node. Full upgrade if memory / load continues to be stable.

Wednesday, Oct 29, 2014 around 1pm PST: 4e21bdb6f to be deployed

~~New parser tests for lang/category/wiki links~~
~~Request and cpu timeouts in the API~~
~~Log process events to LogStash~~
~~bug 57252: Generate HTML5-compliant cite id/about attr values~~

Reverted deploy after running into stuck processes on a few nodes.

Monday, Oct 27, 2014 around 1:30pm PST: Deployed 617e9e61

Tweaks to logging
Improvements to paragraph wrapping to skip over link and other rendering transparent tags
bug 72487 More robust handling of <td>s split across top-level pages and templates

Wednesday, Oct 22, 2014 around 1pm PST: Deployed 2a8dc85c

Send logging events to LogStash

Monday, Oct 20, 2014 around 1pm PST: Deployed d4567e9f

bug 71361: Leave link tags out of p-wrappers wherever possible
bug 68395: Tweaks to p-wrapping around formatting tags
bug 72136: Comply with the 'body' API parameter

Thursday, Oct 9, 2014 around 1pm PST: Deployed 644071d2

Support sourceswiki (multilingual http://wikisource.org).
bug 71880: Fix inserting category links from non-parser function extensions.
Remove backward compatibility support for mw:WikiLink/* types
Code cleanup patches: More use of promises in our API

Monday, Oct 6, 2014 around 1pm PST: Deployed 13a53ab3

Code cleanup patches:
- bug 53109: Move data-mw away from manual json attr loading
- Start using promises API
bug 70196: Add categories added directly from extensions and action=parse
bug 71490: Set prop 'wikitext' when calling action=expandtemplates
Improved logging for failed API requests
Reduce cache request timeout to 10 sec for only-if-cached scenario (from 60 sec)

Monday, Sep 29, 2014 around 1pm PST: Deployed deed30b2

Upgrade domino lib to 1.0.18
bug 69240: Make lang-links sol-transparent
bug 66749: Fix the test for DU.isGeneratedFigure
bug 71312: Do not strip empty <tr> nodes if they have html attrs
bug 67540: Load extension CSS modules
Fixes to tokenizer to parse tables one row at a time
Fixes to tokenizer to release backtracking memory asap
Leave sol-transparent tags out of p-wrappers where possible
bug 64901: Fix paragraph-wrapping to match PHP parser + Tidy combo

Monday, Sep 22, 2014 around 1:25pm PST: Deployed ff9476f9

bug 70894: Bug fixes serializing modified wikilinks
bug 70867: Fix production crashers on some wikitionary pages

Monday, Sep 15, 2014 around 1:30pm PST: Deployed b845bff9

Empty auto-inserted nodes that are transclusion markers should not be deleted.
bug 67452: Delete empty li and tr nodes found in transclusion content.
Indicate the revision in a content-revision-id header
bug 70664: Fix paragraph wrapping to not include transclusion markers where unnecessary.
Edge case fix of DSR computation for fostered nodes

Monday Sep 8, 2014 around 1pm PST: Deployed ce108cb5

Additional CSS classes added to Parsoid HTML <body> elt
bug 67863: Treat {{!}} as a magic word
bug 70370: Fix bug parsing certain extensions found in Flow content

Wednesday, Sep 3, 2014 around 1pm PST: Deployed 78e55c6b

Upgrade of html5 libraries
bug 58184: Suppress --!> as a comment closing tag in browsers
Allow pipe and exclamation point in attribute values
Improved serialization for absolute links
Handle local interwiki links

Monday, Aug 25, 2014 around 1pm PST: Deployed 5b5a5ed5

bug 69331: Represent <section> tags as invisible meta tags
bug 69238: Better handle multiple empty attribute values
bug 69190: Pass title to action=parse requests for extensions
bug 68421: Handle empty template call {{ }} more gracefully
bug 68013: Add title attributes to wikilinks
bug 68112: Additional fixes for template expansion failures found in production logs
bug 69827: Fix regression: support nested <ref> tags once more
Fixes to index page
A bunch of code cleanup

Wednesday, Aug 20, 2014 around 1pm PST: 13c31fc8 (deployment abandoned)

~~bug 69331: Represent <section> tags as invisible meta tags~~
~~bug 69238: Better handle multiple empty attribute values~~
~~bug 69190: Pass title to action=parse requests for extensions~~
~~bug 68421: Handle empty template call {{ }} more gracefully~~
~~bug 68013: Add title attributes to wikilinks~~
~~bug 68112: Additional fixes for template expansion failures found in production logs~~
~~Fixes to index page~~
~~A bunch of code cleanup~~

Not deploying today since we found some regressions -- probably harmless, but requires more investigation. We'll fix this and deploy on Monday.

Wednesday, Jul 23, 2014 around 1:30 pm PST: Deployed 47d4bc83

bug 67857: Dont add empty content blocks in transclusion data-mw object.

Monday, Jul 21, 2014 around 1pm PST: Deployed 1c9277d6

bug 49076: Set page title based on DISPLAYTITLE usage
bug 58930: Update pipe trick support to match PHP parser behavior
bug 68090: Bug fix in parsing of interlanguage links

Wednesday, Jul 16, 2014 around 1:20pm PST: Deployed 060dcb54

bug 57670: Fixes for backtracking in the tokenizer that improves handling of pathological parsing scenarios.

Monday, Jul 14, 2014 around 1pm PST: Deployed d51e6409

Support for HTML <rtc> tag
Support for "extra language links" (added to core as part of bug 32189)
bug 67768: Fix for post-edit corruption in some invalid wikilink scenarios

Wednesday, Jul 9, 2014 around 1pm PST: Deployed 1632288d

Fixes to nowiki escaping in indent-pre content
bug 67451: Fixes to namespace handling in relative titles
bug 63799: Fixes to trailing newline migration DOM pass + fixes to handling of table-like wikitext outside tables
bug 65869: Fix serialization of new interwiki-like links to local-wiki

Monday, Jul 7, 2014 around 1pm PST: Deployed 8ef7b6fe

bug 50122: Fixes excess whitespace in some infoboxes.

Wednesday, Jul 2, 2014 around 1pm PST: Deployed 6afcb8df

bug 67237: Fix citation numbering issue
Enable parsoid on wikimania 2015 wiki

Monday, Jun 30, 2014 around 2pm PST: Deployed 0b365d51

Additional CSS ResourceLoader modules added to Parsoid header
bug 66993: Fixes to handling of lists/headings that following </*include*> tags
bug 66489, bug 66498: Correctly handle table-like wikitext (|, |-, etc.) outside wiki-tables

Wednesday, Jun 25, 2014 around 1:30 pm PST: Deployed 4ef9d6be

bug 67013: Recognize <var> and <rb> as html elements
Allow comments and spaces before table lines
bug 66686: Empty line with comment eats trailing nl token
Small tweaks to indent-pre handling in the serializer

Monday, Jun 23, 2014 around 1pm PST: Deployed 392435a2

bug 66514: Fixed bug in nowiki-escaping of magic words
Additional tweaks and improvements to selective serializer

Wednesday, Jun 18, 2014 PST: Deployed 88a61f81

bug 52618: Suppress <nowiki>s for table WT strings outside tables
bug 52784: Add nowiki protection around quotes adjacent to I/B tags
bug 66341: Strip unsupported table tags during serialization
Cleanup, fixes and improvement to serializer around handling of leading white-space on new lines.

Monday, Jun 16, 2014: deploy cancelled

Deployment cancelled to investigate issues found in testing.

Wednesday, Jun 11, 2014 around 1pm PST: Deployed 3de0dba1

bug 56870: DOM support for DISPLAYTITLE magic word.
bug 52360: Invalid links in HTML are serialized to MediaWiki:Badtitletext message.
bug 53228: Escape nowiki when combined with other wiki markup.
Some serialization tweaks.
More CSS tweaks for Parsoid HTML.

Monday, Jun 9, 2014 around 1 pm PST: Deployed 9b673587

bug 51245: Mediawiki and Parsoid CSS styles linked to Parsoid HTML.
bug 62402: Allow links with angle brackets after anchor.
bug 52641: Accept !! in table data.
Parsoid enabled on outreach wiki and wikimania2014 wiki.

Wednesday, Jun 4, 2014 around 1pm PST: Deployed 165a2042

bug 65871: Fix nowiki escaping bug in template args during serialization.
Fix to tokenizer to better handle table / indent-pre interactions.

Monday, Jun 2, 2014 around 1pm PST: Deployed 04a4bf2b

Bug fix in upright handling for images.
bug 64025: Fix crashers handling pre-like strings.
bug 57670: Fix out-of-stack crashers on some wiki pages.
Additional selser tweaks.
Additional performance tweaks.

Wednesday, May 28, 2014 around 1:15pm PST: Deployed a234af8c

Bunch of tweaks to the selective serializer.
Performance fixes to enable parsing humongous pages (bug 65812).
Bug fix to the XML serializer around handling of HTML <pre> tags.
bug 64113: New empty ExtLinks shouldn't be converted to interwiki.

Wednesday, May 21, 2014 around 1pm PST: Deployed 95929801b

bug 64750: Replace space with underscore in namespace links
bug 60411: Bug fix in nowiki-ing of ";" chars
Improved serialization of new language links.
Support protocol-relative urls.

Monday, May 19, 2014 around 1pm PST: Deployed 3ac048d7

bug 64300: Deal with <nowiki/> escaping around url and other magic links (RFC, PMID, ISBN).
bug 65299: Accept <br/ > and record info in data-parsoid.
Some edge-case improvements to template parsing (see commit summary of https://gerrit.wikimedia.org/r/#/c/133506/).

Monday, May 12, 2014 around 1pm PST: Deployed d1c778ea

Bug fix nowiki-escaping transclusion args.
Improvements to accuracy of DSR information -- eliminates some template wrapping errors.
Eliminate crashers when attempting parse of deleted revisions.
Edge case tweaks to serializer.

Wednesday, May 7, 2014 around 1pm PST: Deployed 71f4e884

bug 64954: Fix for production crashers (edge case).
bug 63642: Fixes bad nesting of formatting and figure elements.
Other minor fixes in the tokenizer.

Monday, May 5, 2014 around 1pm PST: Deployed f2f1f1d7

bug 64747: Additional fixes to template encapsulation code based on production crashers.
Handling of empty redirects (edge case bug).
bug 60517: Upgrade tokenizer (pegjs) from 0.7 to 0.8 -- required lots of tweaking and fixing of tokenizer.
Simple upgrades of other libraries (See https://gerrit.wikimedia.org/r/#/c/130992/)

Thursday, May 1, 2014 around 9:20 am PST: Deployed 5e05c585

A whole bunch of performance tweaks.
bug 64017: Last set of fixes to template encapsulation code.
Use handlebars for ParsoidService views.

Monday, April 28, 2014 around 1pm PST: Deployed cab9348e

bug 64017: Bugfix merging nested template ranges (caused by fostered content in tables) + other fixes.
Logging: Suppress stack traces for warnings
Several link handling fixes
- bug 49688: Handle unescaped single quotes in urls
- bug 63430: Correctly handle multiple # chars in links
- bug 63947: Serializer: handle full stops in link target
- Serializer: Underscores not converted to spaces for interwiki links
- Several other fixes (see https://gerrit.wikimedia.org/r/#/c/126853/ for more)

Wednesday, April 23, 2014 around 1pm PST: Deployed 9c99b0be

Fix oldid logging with error/fatal log entries.
bug 64017: Fix bug merging overlapping template ranges (caused by fostered content in tables)

Monday, April 21, 2014 around 1:45pm PST: Deployed ec51e5d1

bug 63979: Accept comments in eofl position
Support comments before table lines
Improved handling of "bogus" image options

Wednesday, April 16, 2014 around 1:30pm PST: Deployed 0bccf02c

bug 62341: Serialize links with wikitext chars correctly (ex: foo''bar)
bug 62258: Match fixed PHP behavior for framed images with a height specification
bug 63790: Multiple commits to fix crashers found in RT testing
- Accept entities in ref attributes
- Improvements to wrapping of fostered transclusions

Monday, April 14, 2014 around 1pm PST : Deployed fba548cb

bug 63636: Handle multiple colons in titles in subpage-supporting namespaces
bug 60754, bug 62743: Improvements to serialization of interwiki links
bug 51090: Fix parsing and serialization of invalid wikilinks
bug 63790: Fix some edge case template encapsulation scenarios

Wednesday, April 2, 2014 around 1pm PST : Deployed 33471172

bug 63119: Improved serialization of empty i/b nodes.
bug 62647: Fix serialization of headings, etc. after categories.
bug 60829: Accept multiple comments in start-of-line context (headings, etc.).
bug 62834: Accept multi-line comments after headings.
Accept comments in template targets.
Sanitizer fix for handling protocols like news: and javascript: (no security issue, validation happens elsewhere too)

Monday, Mar 31, 2014 around 1pm PST: deploy canceled

Canceled deployment to investigate issues caught in testing.

Monday, Mar 24, 2014 around 1pm PST: Deployed fa03dd20

bug 62805 Dont generate NaN dimensions after edits.
bug 62716 Fixed bug in detecting unresolvable tpl targets.

Wednesday, Mar 19, 2014 around 1pm PST Deployed ff8c49e9

Improved connection timeout handling
bug 62663 Handle non-string extension attribute values
bug 62751 Allow scaling of Vector images

Monday, Mar 17, 2014 around 1pm PST Deployed d0f0080a

Support for manual thumbnail option (thumb=) on images.
Roundtrip empty image attributes.
bug 62323: Improvements to RT-ing of fostered content.

Thursday, Mar 13, 2014 around 4pm PST Deployed 004c7acc

Redeployed.

Wednesday, Mar 12, 2014 around 1pm PST Deployed 004c7acc

~~bug 62500 Parse and roundtrip invalid image options~~
~~bug 62501 Fix image up-scaling for 'format unspecified' images~~
~~A bunch of code cleanup.~~

Because of a bug in the deployment system, the deployment did not happen and Parsoid remained stuck at 98936e7a according to http://parsoid-lb.eqiad.wikimedia.org/_version.

Monday, Mar 10, 2014 around 1pm PST Deployed 681f7b8d

~~New logging framework deployed with improved error reporting to production logs.~~
~~bug 62323 Eat > and [ in table / tr attribute names -- improves parsing / serialization of pages with broken wikitext.~~

Because of a bug in the deployment system, the deployment did not happen and Parsoid remained stuck at 98936e7a according to http://parsoid-lb.eqiad.wikimedia.org/_version.

Monday, Mar 3, 2014 around 1pm PST Deployed 98936e7a

bug 61770 Treat all block tags identically in pre-handler
bug 62025 DSR computation: Properly handle <ref> tags nested in <references>.
New <references> tags are now serialized on their own line.
More liberal parsing of broken table and table-row attributes in wikitext.
Fixed regression dealing with fostered text nodes from tables.

Wednesday, Feb 26, 2014 around 1pm PST Deployed 804ead03

bug 52618 Emit | chars outside tables as | text
bug 48664 Handle multiple conflicting image options properly
bug 61928 Handle templated image options in inline images
Bug fixes in pre-handling and DSR output.

Monday, Feb 24, 2014 around 1 pm PST Deployed 51c71eb0

Enabled CORS on all API endpoints.
bug 51628: Support trailing 'pxpx' in image size options.
bug 48664: Correctly handle duplicate options in image wikitext.

Wednesday, Feb 19, 2014 around 1 pm PST Deployed 76e9b661

Additional fixes for link trail / template interaction.
bug 60413 Support link trails for interwiki links.
bug 51921 Allow template attributes for image attributes.
bug 49400 Support image options that have the "|" char in them.
bug 54803 {{largethumb}} template on nlwiki pages handled properly: parses as expected and serialized properly.

Wednesday, Feb 12, 2014 around 1 pm PST Deployed 96c12747

HTML PRE tsr calculation fixes
Several clean-up and refactor patches
Error logging clean-up
bug 60752 Further fixes for link trail / template interaction

Monday, Feb 10, 2014 around 3 pm PST Deployed 4b751fd

bug 59674 Emit non-piped links for edited redirects
bug 60752 Handle linktrails/prefixes correctly for templated links
bug 51262 Correctly render p-tags in blockquotes

Thursday, Feb 6, 2014 around 12:30 am PST Deployed 52d09c4

Deployed after failed code update of Feb 3, 2014 was fixed.

bug 61082 Add Wikiversity to site list in ParsoidConfig

Monday, Feb 3, 2014 @ 11:30 am PST Deployed 2d663eb

This was the first deploy from our new repository /mediawiki/services/parsoid/deploy. This deploy includes all fixes over the last 6 weeks (from December 16th, 2013).

This deployment saw the following code improvements go out:

Fixes to GC issues that led to memory leaks in node 0.10
First pass over long-standing image handling cleanup.
First steps implementing a logging subsystem in Parsoid.
Code quality fixes to improve robustness of code.

Besides these changes, these specific bugs were fixed.

Images

bug 49942 Wikitext tables inside image captions accepted
bug 48665, bug 48924, bug 49221, bug 56400 Use edited image attributes over original values.

Links

bug 45209 Interwiki links pointing to current wiki parsed as plain links
bug 52855 Update to linktrail/prefix regexp code
bug 56255 Update to linktrail/prefix regexp code
bug 59673 Serialization of new link redirects serialized fixed
bug 50704 {{[[Foo]]}} handled correctly
bug 54877 [[[[Foo]]]] handled correctly
bug 58001 Trailing extlink-like text in wikilink handled correctly [[..[Foo]]]

Refs & extensions

bug 59266 Accept unclosed <references> tag
bug 59782 Multiple <references /> tags handled properly
bug 52430 Non-standard WS in extension tags accepted

Misc Tokenizer

bug 57360 Stray table-end tags ignored in some contexts
bug 59944 ISBN with an X recognition

Wikitext escaping fixes

bug 58469 Url parsing fix during nowiki escaping
bug 57469 Fixes for nowiki escaping of ext-tag like text
bug 51944 Fixes to wikitext escaping of link text

Misc edit/serializer fixes

bug 59672 Fixed serialization of edited magic words
bug 57360 Table end tags always serialized on new lines
bug 50756 Whitespace edits properly recognized

Misc

bug 58335 Improvements to handling of fostered table content
bug 60262 Parsoid binding to specific IP or interface
bug 57394 Parsoid now handles OBJECT element