VisualEditor/Design/Bidirectional text requirements

Preamble

Since MediaWiki in general and Wikimedia projects in particular are used in right-to-left languages, such as Arabic, Pashto, Persian and Hebrew, the projected visual editor must support writing and displaying them easily.

"Easily" in this context means that any user must be able to write mixed right-to-left and left-to-right text as easily as he would do it with pen and paper and see the results while editing.

Although this is true for any text in right-to-left languages, proper support for mixing right-to-left and left-to-right is of particular importance for Wikimedia projects, which are heavily multilingual and frequently cite foreign names.

This document tries to be more functional - to describe what the software should do, rather than technical - to describe how it should be implemented. Some implementation suggestions may creep into the document nevertheless.

The direction property

It must be possible to apply the direction property at several levels:

the level of a content page
a block-level element (usually a paragraph, but possibly also a table cell, and possible a template etc.)
an inline element (effectively ‎, but probably set using a template).

Page-level

The page-level direction property is inherited from the direction of the page content language (see Title::getPageLanguage), which is by default mostly the wiki content language but can be overridden by using the PageContentLanguage hook. As an example, this causes meta:Donor_policy/ar to be displayed correctly RTL because the Translate extension makes use of this hook.

However, it is not possible to override the language through the wiki itself, which is usually addressed by manually applying <div dir="(ltr|rtl|auto)" class="mw-content-(ltr|rtl)"> to the whole page. (See bug 28970.) Having the ability to set it per-page through the wiki is particularly useful for current Wikimedia sites like Meta, Commons, Outreach, Strategy etc., for "embassy" pages in various projects, and for other purposes.

Thoughts

UI idea: this can be an option selectable from a dropdown list placed somewhere in "page properties" in the editing mode (is there such a thing now?)

Another barely related idea: everybody hates that stuff like categories, interlanguage links, DEFAULTSORT etc. are part of the page source. These can also be changed in something like "page properties".

Implementation: This property should probably be stored in the database as a column in the page table. If it is defined, 'mw-content-ltr' or 'mw-content-rtl' should be used accordingly.

Language: It is not directly related to directionality, but it may be useful to set the language (lang attribute) at the same levels. Direction may also be set automatically according to the language.

Paragraph-level

The paragraph-level direction property is inherited from the direction of the page, but it must be possible to override it.

A demonstration of writing left-to-right and right-to-left in LibreOffice. Notice the toolbar buttons and the dropdown at the bottom of the dialog. Click to enlarge.

In existing software there are usually three ways to do it:

Pressing Ctrl+⇧ Shift (see #Keyboard interaction).
Pressing a button in the toolbar. It usually looks more or less like this: ¶◄ (see screenshot).
Selecting an option in a "Format paragraph" dialog (see screenshot).

As far as the internal implementation goes, adding a dir attribute to the ‎ element should be enough.

Several paragraphs at once

Pending question: Should the setting of direction for several paragraphs at once exist? Something like:

<div dir="rtl">
<p>RTL paragraph 1</p>
<p>RTL paragraph 2</p>
<p>RTL paragraph 3</p>
</div>

It would be comparable to a section in OpenDocument.

For display purposes setting direction per paragraph is enough, but applying this to several paragraphs may be useful for semantic markup, for example to indicate a long quote.

Inline

This is needed for writing a string with directionality X inside a paragraph with directionality Y.

Common scenarios:

Scenarios	correct	incorrect
An English brand in the Arabic Wikipedia	Yahoo!	Yahoo!
An English brand in the Arabic Wikipedia	7 Up	7 Up
A Hebrew brand in the English Wikipedia	וואלה!	וואלה!

Most likely this would be an HTML ‎, but it can also be ‎<bdi> (in HTML5), ‎<code> and other elements.

In other rich text editors: OpenDocument doesn't support this feature natively, but only through the use of control characters, such as RLM and RLE/PDF. Microsoft Word seems to try to support it by assuming the inline text direction according to the language of the current keyboard layout, which is rather intuitive and user-friendly, but it doesn't offer any other way to control it afterwards, which may be confusing.

Implementation idea: A simplistic way to let users do this would be to create a template like this: {{{1}}}. Since the Visual editor is supposed to be able to handle templates, this will require very little coding. However, this would not be completely robust, because the templates must be actually created manually after installing MediaWiki with the Visual editor, while it would be better to have it installed automatically. A structured way to manage templates, for example by global transclusion or a "standard template library" - ideas that are occasionally discussed at wikitech-l - would make this more robust.

Keyboard interaction

Direction setting

The usual keyboard shortcut in Windows and Linux for setting direction is Ctrl+⇧ Shift. Left Ctrl+⇧ Shift sets the direction to LTR and right Ctrl+⇧ Shift sets the direction to LTR.

In rich text editors such as LibreOffice and Microsoft Word it sets the direction of the current paragraph or of all the selected paragraphs. In plain text editors such as Notepad and browser textareas it sets the direction of the whole text, as there's no robust way to set the direction of a single paragraph in plain text (RLE/PDF is not quite robust).

Notable exceptions:

Firefox: Ctrl+⇧ Shift does nothing. Ctrl+⇧ Shift+X, no matter at which side, switches the current direction (bidi.browser.ui must be set to true so this would actually work). It seems to be so because DOM doesn't define precisely what to do with Ctrl+⇧ Shift by itself. There's a very old request to change it to Ctrl+⇧ Shift (Mozilla Bug 98160 - Replace Accel+Shift+X (Switches text direction) with more intuitive keyboard shortcuts: Ctrl+Left/Right Shift on Windows).
Notepad++: Ctrl+Alt+L, Ctrl+Alt+R.
The commonly used rich text editor in GMail, which is actually a complete HTML iframe, doesn't provide any way to change paragraph direction with the keyboard. It can be changed using a toolbar button (which only appears if the interface language in RTL).
In supporting text editors on Mac OS X the shortcut is Ctrl-Command-←/→. (Must be turned on in system keyboard settings.)

Since Ctrl+⇧ Shift is the common shortcut, the MediaWiki visual editor should use it in a fashion similar to LibreOffice and Word - to set the direction of the current selected paragraphs. It should also be possible to interact with the client system as much as possible and allow the user to use the shortcut to which she is accustomed in the software.

Important question: Should Ctrl+⇧ Shift be used to set the direction of a selected inline element? The intuition says that it should only work for paragraphs, but maybe user testing will prove that users like it. (And maybe Ctrl+⇧ Shift+X should be used for that?..)

Movement

It must be possible to use the right and left arrow keys intuitively. While the cursor is in words in the main language of a paragraph, the cursor should move in the direction of the arrow. The hardest decision to make is how should the cursor move in a text in the other direction, for example in a Hebrew word within an English paragraph. In different programs it works differently:

program	direction
Firefox	visual (see notes)
Chrome	visual
MSIE	logical
MS Word	logical
LibreOffice	logical
Notepad	logical
more examples would be nice	especially from Mac and Linux

Firefox notes:

In Firefox this could be changed by setting bidi.controlstextmode in about:config, but this was disabled.
The movement is visual also when using the visible caret (enabled in the accessibility settings).

Text selection always works in the logical order, because visual selection will cause non-contiguous text to be selected.

The movement order should probably be logical, since it is consistent with the selection order and more common (and probably easier to implement, too).

Another consideration is Ctrl-Arrow, which makes the cursor jump from word to word. It should be obvious that its direction must be identical to that of the regular movement arrows, but in Chrome it's implemented incorrectly (Chromium Issue 10741: Ctrl + Right/Left combination moves cursor to opposite direction between words in RTL text box).

(Also Ctrl-Shift-Arrow, Ctrl+Del etc.)

Values for direction

The possible values for direction in HTML are "ltr" and "rtl". HTML5 adds "auto", which, according to the specification, applies the direction of the first character with strong directionality to the whole element.^[1] It already works in Webkit (and even used in some elements that MediaWiki outputs) and will probably work in Firefox in mid-2012. It doesn't work in IE9.

HTML4 prohibits automatically setting the direction according to language. In standard HTML the default direction is "ltr" and it will remain so even if the language is explicitly defined to a code of an RTL language - is "ltr". HTML5 is not different in this regard. It is not clear why is this so, but the probable reason is that there are thousand of language codes, their relationships with scripts is not completely defined and so it's impossible to rely upon webmasters to use correct codes and upon browsers to determine correct direction.

Despite this, there is no problem for MediaWiki to determine the direction according to the language and write it explicitly to the output HTML. In fact, this is done already, especially since 1.18. MediaWiki also strives to use standard codes (although there are deviations from this rule in the current projects). Furthermore, the Visual Editor should encourage specifying the language using the lang attribute, by making it possible and easy (see Visual editor/Internationalization requirements). The Visual Editor should, therefore, provide the following values for direction:

ltr: dir="ltr"
rtl: dir="rtl"
auto: dir="auto" (?)
by language: let MediaWiki determine the direction according to the lang attribute.

It's possible that "by language" should be the default value, but this requires some thought.

Isolation

Bidirectional isolation prevents the directionality of an element from interfering with the surrounding content. This often happens, for example, when an English name appears before a person's birth date in an RTL wiki and when a footnote number appears after a Hebrew word in an LTR wiki.

In Wikimedia projects the foreign text is usually not related to the surrounding content, and it is therefore worth considering making an element bidi-isolated by default (but when - when the language is different? when the directionality is different?).

This can be done both by putting the content in a ‎<bdi> element^[2] or by applying the CSS rule "unicode-bidi: isolate".^[3]

Enabling and disabling the controls

It should be possible to disable the bidirectional controls - buttons, dropdowns, dialogs -, as well as the relevant keyboard shortcuts (Ctrl+⇧ Shift?). It is likely that many people will never use them. For example, someone who writes in an LTR project and never deals with Middle Eastern themes will probably never use them, so they may be unnecessary clutter in the toolbar.

LibreOffice, Microsoft Word, GMail and Google Docs only enable these bidirectional controls if the interface language is RTL or if the user specifically requested it. MediaWIki Visual editor can take a similar path: This should be one of editing preferences, enabled by default in RTL wikis and for users who chose an RTL language in the preferences. It must be possible to enable it in any setup (GMail is buggy in this regard last time i checked).

Usage of control characters

Several transparent control characters are used for controlling directionality:

RLM / LRM - transparent characters with strong directionality. It's easy to understand them as Hebrew and Latin letters, respectively, but invisible. They are used for isolating inline pieces of text with ambiguous directionality, for example a Latin letter from a following number in a right-to-left paragraph. Equivalent to the HTML entities &rlm; and &lrm;.
RLE / LRE - start an embedded inline block of right-to-left or left-to-right text that runs until a PDF character. Similar to /.
RLO / LRO - starts a section in which the Unicode bidirectional algorithm doesn't apply. That is, Hebrew letters will appear left-to-right. Equivalent to the HTML tag ‎<bdo>.
PDF - pop directional formatting. Ends a block of text started by RLE, LRE, RLO or LRO.

There's no practical use for RLE, LRE, RLO, LRO and PDF in text that can be marked up with HTML.

RLM and LRM are currently frequently used in MediaWiki, as there is no other common way to isolate pieces of text with ambiguous directionality. They are often inserted as a template (such as w:he:Template:כ). Implementation of the HTML5 ‎<bdi> tag may diminish the need for them, but currently they are needed.

Other considerations

Offline and PDF export: These features shouldn't be directly affected by this if parsing and rendering stays the same, but they must be tested anyway.
Web fonts.

Footnotes

↑ Draft HTML5 spec - the dir attribute
↑ At the time of writing it is Implemented in Gecko and Webkit, albeit with some bugs, and it is sanitized in MediaWiki.
↑ At the time of writing it is implemented in Webkit; in Gecko it still experimental and has the vendor prefix -moz-isolate.

[1] Draft HTML5 spec - the dir attribute

[2] At the time of writing it is Implemented in Gecko and Webkit, albeit with some bugs, and it is sanitized in MediaWiki.

[3] At the time of writing it is implemented in Webkit; in Gecko it still experimental and has the vendor prefix -moz-isolate.

[1]

[2]

[3]