Parsoid/Parser Unification/Confidence Framework/Reports

Questions to answer for report assessment

• What’s the visual diff score? What will make us confident to deploy based on this score?
• Are there specific extensions critical to Wikitech DiscussionTools that are not yet supported?
• Are there missing functionalities in Parsoid that will impact this deployment?
• What’s the rollback plan in case needed?
• What would be the main concern and reason to postpone this deployment?
• What’s your proposed confidence score?
  • Not confident at all
  • Not confident
  • Somewhat confident
  • Confident
  • Extremely confident

Deployment Readiness for Wikivoyages

For all wikis where Parsoid readviews is being rolled out, some common considerations apply:

• Extension support: All critical extensions are known to be functional on the wiki.
• Testing considerations: Visual diff testing only identifies rendering issues and may not uncover dynamic behavior like javascript functionality - in some scenarios, we may discover regressions post-deploy and aim to quickly fix them as we discover them or rollback Parsoid if appropriate.
• Rollback Strategy: A full rollback is planned if any significant issues are observed post-deployment.
• Functionality Concerns:
  • Kartographer: The wikivoyage projects use Kartographer heavily, so we are paying close attention to interactions Parsoid may have with Kartographer
  • Localization/RTL issues: For non-English wikis, localization of messages will be a key issue (localization of messages from Kartographer in particular). This is also our first Right-To-Left reading order wiki; we don't expect this to be an issue but we will watch carefully.
  • Vertical whitespace issues: A relatively large number of pages display minor vertical whitespace issues. This is primarily due to several heavily-used templates which contain slightly-erroneous wikitext which triggers these issues, and can be addressed by fixing up the responsible templates. In some cases the Parsoid output is actually better than the output of the legacy parser; i.e. Parsoid omits unnecessary vertical whitespace added by the legacy parser.

Wiki	Confidence	Visual Diff Score					Deployment date
		Tested Version	Tested pages	Pixel Perfect Rendering	Vertical WS shifts only	Potential issues
eswikivoyage	High	1.43.0-wmf.20	890	90.79% (808)	98.88% (880)	1.12% (10)
	Remarks: a couple of edits fixed broken wikitext causing diffs. Mostly Vertical White Space and couple of cases that contain minor table padding differences in a couple columns or gallery centering differences where legacy parser is actually worse.
eowikivoyage	High	1.43.0-wmf.20	869	90.45% (786)	99.19% (862)	0.81% (7)
	Remarks: Vertical White Space shifts noted. Gallery rendering issues due to a markup error was fixed. See this page for detailed diff analysis.
fiwikivoyage	High	1.43.0-wmf.20	1035	87.83% (909)	99.03% (1025)	0.97% (10)
	Only Vertical White Space issues and one minor issue with a list item not getting parsed. See this page for detailed diff analysis.
svwikivoyage	High	1.43.0-wmf.20	1154	82.06% (947)	98.61% (1138)	1.39% (16)
	Remarks: T371125 has been identified in the visual diff, but doesn't impact functionality and it's not a blocker to the deploy. VWS issues and extlink parsing problem in some cases, fixed after editing the wikitext in the template. See this page for detailed diff analysis.
rowikivoyage	Very High	1.43.0-wmf.19	623	88.76% (553)	98.72% (615)	1.28% (8)
	Initially, we've identified a difference in the main page that required a wikitext change that can be found here.
cswikivoyage	Very High	1.43.0-wmf.19	403	98.01% (395)	99.5% (401)	0.5% (2)
	We are working on an issue with the error messages displayed by the geodata extension at phab:T372608 but do not consider this a blocker at this time.
hewikivoyage	Confidence in the rollout is very high, although we will take care due to the functionality concerns described.	1.43.0-wmf.14	4,799	53.0% (2,544)	99.3% (4768)	0.7% (31)
	46.3% of pages show only minor vertical whitespace shifts (2,224/4,799). The vertical whitespace shifts are primarily due to the interaction of invisible tags (usually template boundary markers) and paragraph wrapping (T355099/T368719), which the Content Transform Team considers a "known difference" between the parsers. An additional page (1/4,799) displayed a rendering difference due to T368720 caused by a misnested <small> tag. A patch is in progress for this issue. The remaining 0.6% of pages (30/4,799) displayed false positives, such as images not completely loading, which are artifacts of the visual diff process.
enwikivoyage	Very High	1.43.0-wmf.14	24,609	90.9% (22,363)	99.7% (24,524)	0.3% (10)
	The remaining 0.3% of pages (85/24,609) displayed false positives, such as images not completely loading, which are artifacts of the visual diff process.
bnwikivoyage	High	1.43.0-wmf.20	811	97.04% (787)	98.64% (800)	1.36% (11)
	Mostly VWS and edge cases. Initially, some map banner differences were identified and reported at T373400, but after deep investigation it didn't become a blocker, but a known issue.
hiwikivoyage	High	1.43.0-wmf.20	949	91.04% (864)	98.63% (936)	1.37% (13)
	Initially, some map banner differences were identified and reported at T373400, but after deep investigation it didn't become a blocker, but a known issue. We also identified a few pages with hidden tracking category, also considered not a blocker and will require further investigation.
pswikivoyage	High	1.43.0-wmf.20	800	95.0% (760)	98.38% (787)	1.62% (13)
	Table markup error leads to different error handling -- recommend fixing page. Fixed with this edit. Gallery rendering differences because of a markup error. Fixed with this edit.
trwikivoyage	High	1.43.0-wmf.20	875	89.94% (787)	98.17% (859)	1.83% (16)
	Initially, some map banner differences were identified and reported at T373400, but after deep investigation it didn't become a blocker, but a known issue.

Deployment Readiness for Wikitech DiscussionTools

Visual Difference Score Assessment

Current Status:

• Pixel Perfect Rendering: Achieved for 95.7% of approximately 6250 talk pages tested.
• Minor Issues: 99.9% of pages show pixel perfect or only minor vertical whitespace shifts.
• Known Differences: 0.1% of pages have known differences (not planned for fixing).
• Critical Differences: None.

Considerations for Deployment:

• Reliability: The high percentage of pixel-perfect rendering (98.55%) is a strong indicator of the system’s reliability.
• Pending Reassessment: None
• Blocker Evaluation: None

Extension and Functionality Support

Critical Extensions:

• Assumption: Extensions critical for Wikitech DiscussionTools are presumed to be identified in the visual diff process.

Functionality Concerns:

• Login vs. Logout Issue: None
• Legacy Parser Compatibility: A strategy is needed to ensure that deployment of the new system does not disrupt the recording of metadata by the legacy parser.

Rollback Strategy

• Procedure: A full rollback is planned if any significant issues are observed post-deployment.

Main Concerns and Potential Delay

• Current Stance: As of now, there are no major concerns that would necessitate postponing the deployment.
• Dependent Factors: None.

Proposed Confidence Score for Deployment

• Score: Confidence in the rollout is extremely high.