Topic on Talk:2017 wikitext editor/Flow

There's a debate at Talk:Wikimedia Product which seems to be going nowhere because it is not clear what the planned behaviour of the new editor is with respect to previewing.

Is it planned that the new editor should be capable of rendering what the current editor renders, and in the same way? That's a design decision that will have been taken early on. However the WMF interprets "Agile" (and I think the prevailing interpretation is not an entirely productive one), the answer to this cannot reasonably be, let's write some code and see whether it does or not. That decision, and those differences, will have to have been made clear early on, so that the community can decide whether it is a blocker (see discussions around Technical Collaboration Guideline/Community decisions) and if it is agreed by the community that they are willing in principle to accept differences in behaviour, and agreed the scope and scale of the differences proposed, then there needs to have been a discussion leading to a clear description of what will and will not be rendered the same way by the two software systems. It would now be useful to see the records of those discussions and those decisions.

In the case of the specific debate I mentioned, there is a non-meeting of minds because these issues have not yet been surfaced, and the the resolution has to be that if the new editor is supposed to render HTML in the same way as the current one, then the behaviour reported is simply wrong. If the new editor is supposed to render some input the same and some differently, then the behaviour here may or may not be wrong, depending on the proposals for the scope of the differences to be tolerated between the two rendering engines.

So: is the new editor supposed to render the same code in the same way as the old one? If not, what is the scale and the scope of the difference? Simple questions. I hope someone can publish the clear, definitive and agreed answers.

+1, and thanx Rogol for bringing the discussion from Talk:Wikimedia Product to here.

I think accurate previews is a critically important issue for community acceptance of the new editor. Experienced editors expect the wikitext editor to give accurate previews, and new editors would be particularly harmed by inaccurate previews. A critical reason wiki has been successful is because a new user can just jump in, try stuff, and preview to see how it works. Inaccurate previews will confuse them and undermine their ability to figure out what's going on, and why.

The current version of the new editor clearly uses Parasoid for previews. This results in many random inaccuracies in how preview are rendered. It seems obvious that the wikitext editor should use the same (PHP) render engine that article pages use, just like the current wikitext editor does. This automatically gives accurate previews. This also ensures that previews and article views always stay in sync "for free" when any update is made to article rendering.

Can someone from this team answer whether you'll fix this? If there is any question whether this is a significant issue, I can open a community process to get you feedback on whether the community considers this an important blocker.

We continue to iterate both parsers towards the same output for the same input. Any differences between them should be filed in Phabricator so that they can be fixed.

Parasoid is clearly not ready for this purpose. It can't even display red/blue/black/external links properly, sometimes it can't track refs properly, and there's an unknown but vast number of random things it gets wrong. I'm just one editor making occasional exploration of the issue, and I continually find new random issues in Parasoid renders.

So back to my question, is the WMF willing to switch New Editor to use the article-parser for previews?

If there is any question whether this is a significant issue, I can open a community process to get you feedback on whether the community considers this an important blocker. I expect the community is going to want any New Wikitext editor preview to be as accurate as the current wikitext editor preview. I expect the community won't want new editors getting a sabotaged/broken wikitext environment.

I think this is going to be a blocker.

@Jdforrester – This is a fine aspiration, but I find it hard to believe that they will ever be completely and reliably identical. The issue is larger than the occasional unintentional differences and synchronization errors. The Visual "WYSIWYG" editor has a fundamental requirement to render something amenable to editing. "Preview" has a fundamental requirement to render exactly what will be seen when the page is saved. These are different requirements.

In the visual editing environment, if What You See Is sometimes not exactly What You Get, whether due to incompleteness, or error, or some practical compromise by design, it is a feature of that editing environment, and needs to be dealt with or accommodated or endured in that environment. The purpose of a "Preview" function is to step away from the editing environment and display what you will actually get when the page is saved.

Whatever advantages may be perceived in using the Visual "WYSIWYG" engine in both environments, they cannot be allowed to override the fundamental purpose of a "Preview" function. It is not to view what you would see in another editing environment, but to view what you will actually get in the reading environment.

For a Preview function to fulfill its fundamental purpose it is clearly logically sufficient to use the same engine that renders saved pages in reading mode. I think it is also pretty obviously logically necessary because in the real world, unlike the world of aspirations, it must be expected that differences can and will creep in between What You See when editing and What You actually Get upon saving.

Any other approach, however well intentioned, must miss the mark and ought to be a blocker.

Ping Jdforrester (WMF). Can we get a response here?

Thanks for that clarification. It has been said elsewhere that "the goal is to have a single parser for all editing tools and all platforms" which is a stronger assertion (but consistent with it of course). Is it currently intended that Parsoid will be that product? Please indicate the location where we can see the high-level plan (roadmap, stragegy, whatever you like to call it) which makes things like this clear, so that we don't waste your time asking questions like this.

Well, the principal declaration was in this year's annual plan (https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2016-2017/Final, specifically " we will unify around a single Wikitext parser for both reading and editing" and "We will modernize the wikitext format to be more powerful and predictable, using a robust testing system to guide our work. This will include minor syntax changes and improved templates, making them simpler by restricting the technical impact on the output of the page.".

Individual work against this is generally tracked in the quarterly reviews and Phabricator. The current major area of work is replacing Tidy and better native support for various things like <gallery> so that they work identically.

Thanks, although that declaration is very high-level indeed. The new Technical Collaboration Guideline implies that lack of a dedicated project or product page is a blocker ipso facto. Presumably such a page will be created for the single parser project when those Guidelines come into force?

I thought that this page is that. In what ways do you think it fails to be a page dedicated to the project? What is missing that we can improve?

It seems unlikely that a page titled "2017 wikitext editor", and on which the word "parser" does not appear, is a page dedicated to the project of unifying around a single wikitext parser. It may well be a page dedicated to a new editor, and the new editor may well be connected with the unified parser project, but that does not make this page the page dedicated to the unified parser project.

Oh, sorry, I didn't understand your meaning. I think you want Parsoid, a project that's been underway since 2011.

No, I am talking about the parser unification project or single parser project, which, as you pointed out was foreshadowed in the 2016/2017 annual plan, and which, according to your comment, is currently documented in a disparate collection of internal reports and Phabricator tasks. It is not, as far as I can tell, documented in a project page or plan of its own, which is odd for a project specifically mentioned in the annual plan. Parsoid is a piece of software which as you point out has been around for some time (and which I am aware of, thank you very much). It is a parser but not a parser unification project. Parsoid might be the parser around which you propose to unify, I don't know. There needs to be a page explaining, for the benefit of the participants in this project, in related projects, and the developer and user communities at large, what this parser unification project is, what it aims to do, how in broad terms it aims to do it, who is involved, what steps are being taken, what the timeline is, and all the other information that is needed to get a satisfactory community engagement around it. This is not a matter of satisfying my personal curiosity, but a matter of good community engagement. Indeed, as I have pointed out, the Technical Guidelines under discussion right now suggest that the absence of this sort of communication is a blocker in itself.

So please either point to the page that fulfills that role for the parser unification project, or, if there is no such page, just say so, and say what steps you propose to take to ensure it is delivered.

The "parser unification project", as you put it, is part of the work of the Parsing team and captured on Parsoid, as I said.

I am referring, as I am sure you are perfectly well aware, to Meta:Wikimedia Foundation Annual Plan/2016-2017/draft#Program 4: Maintain and improve content creation and editing tools which states as "Goal 1: Maintain and incrementally improve current content creation and curation interfaces" / Objective 1 In support of these changes, unify around a single Wikitext parser for both reading and editing. Do you seriously challenge my calling this the parser unification project? Do you seriously maintain that as Product Manager of the Editing Department, you are not thoroughly familar with the nature and content of this Objective? Do you deny that you could provide more details if you wanted to?

More seriously, there is no mention of a proposal to unify around a single wikitext parser on the page Parsoid and I fail to understand why you suggest there is. Indeed, I am unaware of any more detailed description in the public arena of this component of this objective than the single sentence I have quoted above. If you can point to such a document then please do so, and if there is no such description then please say so. Simply repeating an unhelpful comment is unhelpful.

Rogol, the Technical Collaboration Guideline is a set of recommendations. Even when published officially, not having a product page is not an automatic blocker because the TCG is not a part of a product's development process - it's suggested best practices. Please don't try to use it against product teams in this manner.

Do understand, it is a best practice to publish this information on a wiki page, it's great for community engagement, but my point is that I don't want the TCG to be used as points against product teams. It weaponizes what can otherwise be a positive, cultural influence internally. I'd like to focus on doing things better with it.

Firstly, I'm baffled by what practical value you expect the TCG as a set of recommendations or suggestions for best practice to have if you expect no one to point out when a project is not conforming with them. I too am anxious that the WMF and the community should do things better together, which is why I am disappointed to hear that this new project is starting off in a way that is not apparently consistent with what will shortly be proposed as best practice, but far more importantly, in a way which runs the risk of failing to involve the community of editors and readers who are going to use the product when it arrives. These are not disjoint issues. The object, which I applaud, of the Guidelines, is to increase the level of productive engagement. To use the language of "weaponization" is unhelpful and I am disappointed that you should have done so.

Secondly, please would you be so kind as to give the answer to my question, which I presume to be a simple "yes" or "no", as to whether the parser unification project or single parser project, as foreshadowed in the 2016/2017 annual plan, currently has a single project page or plan which can be exhibited to editors and readers?

Thirdly, as a follow-up, if the answer is currently "no", please would you be so kind as to say whether it is proposed to provide one, and if so what steps are being taken to deliver it? On the other hand, if the answer is "yes", please be so kind as to indicate its location.