Jump to content

Requests for comment/HTML templating library

From mediawiki.org
Tracked in Phabricator
Task T379
Request for comment (RFC)
HTML templating library
Component General
Creation date
Author(s) Kaldari
Document status accepted
for mustache templates, see IRC discussion
Approved -- Tim Starling (talk) 21:24, 22 October 2014 (UTC)[reply]
See Phabricator.

Yes MediaWiki supports mustache templates, both client- and server-side; see Manual:HTML templates.

Several MediaWiki extensions make use of JavaScript or PHP HTML templating libraries. It would be ideal to standardize on one library and add it into MediaWiki core. This is similar to the situation that existed regarding CSS languages before LESS support was added to core and made the standard.

We decided to standardise on Mustache, with the view to revisit this decision later.

Note: This is mostly unrelated to the use of MediaWiki templates in wikitext (such as the {{RFC }} in this article). Templating here builds the HTML of a web page, not the content within it.

Progress steps

[edit]

(2014-10-23) In an IRC RFC meeting, it was resolved that Mustache be used as the recommended template processor in core and extensions, for both JavaScript and PHP code.

(2015-03-01) Gerrit change 181708 added the lightncandy PHP template parser to core/vendor, and Gerrit change 187728 added a TemplateParser class providing a server-side interface to cachable Mustache templates that are dynamically compiled by lightncandy. Gerrit change 180647 adds ResourceLoader support for client-side JavaScript templates to core, along with the mustache.js library.

Justification

[edit]
  • Better code readability, editability, and portability
  • Separate code from markup
  • Avoid jQuery spaghetti
  • Avoid loading multiple JS libraries from different extensions
  • Make it easier to build complex and good looking admin interfaces

Existing implementations in MediaWiki extensions

[edit]

Templating library options and considerations

[edit]

It would be best if we could choose a library that has both JS and PHP implementations. That will make it easier to share/port code between client-side and server-side, and eliminate the need to learn two different syntaxes. It should also be a library that is lightweight enough to use on mobile, but flexible enough to meet the needs of diverse applications.

Requirements

[edit]
  • Both a PHP and JS implementation
  • if/else
  • array handling
    • foreach
    • detect an empty array
  • Secure by default
    • Attribute sanitization (no XSS in href, src, style etc)
    • HTML escaping and DOM balancing
    • Escape hatch for things that are known to be pre-escaped, but this should be easy to audit, ideally by checking a single place. Ideally it is not necessary to read all templates for this.
  • Extensible enough to accommodate MediaWiki i18n system
    • Should either be a custom implementation of the library or a predefined function/filter
  • No huge client-side footprint
  • Dumb (the less logic the better). Some template languages, for example jinja2 have a concept of filters where computation happens in the template itself. In my opinion this is unnecessary and can lead to complicated unreadable templates. Data should be preprocessed in PHP before being passed to a template.
    • The opposite position is that simple logical and arithmetic expressions in conditions and formatting filters can save a lot of very template-specific code in the controller. With some very basic added functionality, many simple templating tasks can actually be performed just with a template and the model. Filters can be a way to implement i18n, currency formatting or pluralization, which are arguably templating task. The power of expressions and filters should however be limited to the bare minimum to preserve portability and avoid unnecessary complexity.
  • Readable - templates should be easy to grasp for someone with basic HTML knowledge. Ideally they are just plain HTML with attributes, so that they can be displayed and edited with HTML tools (think VE). This can enable server-side pre-expansion followed by client-side updates.
  • Commenting

Performance

[edit]

MaxSem did some profiling of both Twig and Mustache on the server side (ported an existing special page to it) and compared the performance with using regular MediaWiki HTML generation. According to Max, the performance characteristics were very similar for all three. For example:

  • Original code 5710ms 50th percentile, 5741ms 90th percentile
  • Twig, uncached: 5693, 5738
  • Twig, cached: 5700, 5741

Mustache's results have approximately the same unnoticeable difference from original code.

CSteipp and Gabriel Wicke wrote a set of scripts to compare different scenarios, and timed the results on ruthenium using PHP 5.3.10, HHVM nightly and Node.js 0.10 (using the runall.sh script). [1]

All times in seconds (best of 50)
Engine Test1 (text + attr interpolation) Test1b (as Test1, attr value random) Test2 (iteration over objects) Test2 (iteration over objects + lambda) Test3 (iteration over strings)
TAssembly (node.js) 0.0420 0.0630 0.6190 0.5010 0.1860
Knockoff (node.js) 0.0530 0.0750 0.6370 0.4690 0.1970
Handlebars (node.js) 0.1300 0.1750 0.6060 1.1450 0.1870
Hogan (node.js) 0.1110 0.1370 0.6690 4.8780 0.6770
Spacebars/TAssembly (node.js) 0.0650 n/a 0.6430 n/a n/a
Mustache (node.js) 0.3570 0.3900 2.5200 3.8260 0.8830
TAssembly (PHP) 1.7434 1.7654 18.6634 81.8341 12.3270
TAssembly (HHVM) 0.5199 0.5317 3.3146 16.7471 2.1257
Handlebars lightncandy (PHP) 0.5395 0.6025 7.0456 133.2025 3.9837
Handlebars lightncandy (HHVM) 0.1919 0.1991 0.8164 1.3319 0.5254
Mustache (PHP) 1.7991 1.8606 11.3725 24.0264 3.6850
Mustache (HHVM) 0.4783 0.5016 1.2869 3.9279 0.6229
MediaWiki Templates (PHP) 1.6926 1.7354 17.0633 n/a 7.4075
MediaWiki Templates (HHVM) 0.5305 0.5147 4.0606 n/a 1.9745
Twig String (No Cache) (PHP) 1.3954 1.4701 6.0498 n/a 3.6746
Twig String (No Cache) (HHVM) 0.5631 0.5137 0.9394 n/a 0.6560
Twig File (No Cache) (PHP) 1.7074 1.7649 6.0364 n/a 3.6477
Twig File (No Cache) (HHVM) 0.6085 0.6324 0.9831 n/a 0.8150
Twig File (Cached) (PHP) 1.7352 1.7783 6.0423 n/a 3.6068
Twig File (Cached) (HHVM) 0.4170 0.4460 0.7116 n/a 0.4475
Handlebars HTMLJS (node.js) 0.5610 0.6540 7.1110 n/a 2.9330
Spacebars/HTMLJS (node.js) 1.1450 1.3080 59.0910 143.3680 42.2020

Security

[edit]

Twig, and a lot of other template engines, use the file system for caching compiled templates, resulting in a possible attack vector that Chris and ops wouldn't like (but not necessarily prohibit entirely): cache is just PHP files that get executed on page views and potentially by maintenance scripts too, so having Apache write something executable is a bit icky. (by default, Twig does not cache [4])

  • It looks like there are hacks to put it into memcached too, or scripts to pre-compile them all like TemplateCacheCacheWarmer.

My (CSteipp) major concern on security is how the template engine encourages developers to code securely (or makes coding securely easy), and how difficult it is to review an application for security that uses the engine.

  • Twig uses something close to htmlentities on all strings by default (notably, "'" => "& #039;")
  • Mustache uses htmlspecialchars, so ' is not escaped (so by policy, attributes can never be quoted with single quotes, which is what we have currently in MediaWiki's templating).
  • Neither Twig nor Mustache have context-aware escaping (to avoid XSS attacks via attributes etc), which MediaWiki's current templating does provide. They also don't balance the DOM, so broken user-supplied HTML or templates can have very non-local effects.

Libraries

[edit]
Library name Implementations Features Shortcomings Notes
Mustache/Hogan/Handlebars PHP, JS Mustache supports if/else (boolean only), array handling, HTML escaping, lambdas (custom filters), and comments. Hogan adds template inheritance.
Handlebars adds alternative control-flow syntax, partials and helpers (which alllow for easy i18n integration). Also handlebars supports server-side template precompilation which is crucial for clientside performance. More merits of Handlebars on talk page.		 primitive "if/else" support (boolean only); doesn't automatically make array indexes available (Mustache) Already in use by mobile (JS only), used by some Wikia extensions.
Flow team has been working with Handlebars (JS and PHP) until Knockoff is available in PHP.
Twig/Swig PHP, JS Has flow control, inheritable templates, filters (callbacks into native code), a sandbox for untrusted templates, HTML escaping by default or off (can then use a filter to escape), support for iterables, complex boolean expressions, and complex math. May be too powerful for its own good. Haven't looked at Swig to see how much of the default behavior is replicated in JS. Templates are cached as compiled PHP objects. Used by fundraising for emails and an unsubscribe mediawiki extension.
Nirvana PHP Templates are PHP files (supporting the same capabilities as Quicktemplate) or Mustache files but are not Objects. Also, we never use the Xml objects to build output. PHP templates are server-side only. Mustache templates can be used in both contexts (MustachePHP). Tightly coupled with the "Controller" class which is also part of Nirvana. Doesn't provide significant benefits over QuickTemplate by itself. Already in use by several Wikia extensions and the Oasis skin.
Smarty PHP
AngularJS JS client-side, node possible with jsdom, PHP would require port Templates are valid HTML files with additional attributes, associated with controllers and bound to a model. Changes to the model are reactively reflected in the view. Good security support with context-sensitive escaping and a balanced DOM. Flexible architecture with dependency-injected services including localization and animation. Directives include client-side form validation and formatting filters. No PHP implementation. 36k compressed, but no other dependencies. Used in Google analytics etc, but no use in MediaWiki so far. HTML with attributes a good basis for visual editing. Markup and expressions strike a good balance between ease of use and portability. Can work well with server-side pre-rendering plus client-side dynamic updates as regular content with attributes can be the template.
KnockoutJS JS client-side, experimental node support. PHP would require (relatively straightforward) port Templates are valid HTML files with additional data-bind attributes, associated with controllers and bound to a model. Changes to the model are reactively reflected in the view. Good security support, easy to add context-sensitive escaping. Ensures a balanced DOM. Simple data-bind attribute that can be extended with custom bindings. 16k compressed. No PHP implementation, but fairly easy to port. data-bind syntax very simple and efficient to process, but somewhat less intuitive than the direct interpolation with expressions and filters in AngularJS. HTML with attributes a good basis for visual editing. Easier to port than AngularJS. Can work well with server-side pre-rendering plus client-side dynamic updates as regular content with attributes can be the template.
KnockOff (see this sub-RFC) JS, PHP Templates are (KnockoutJS compatible) valid HTML files with additional data-bind attributes, associated with controllers and bound to a model. Good security support with context-sensitive escaping. Ensures a balanced DOM. Simple data-bind attribute that can be extended with custom bindings. Runtime 3.2k compressed. data-bind syntax very simple and efficient to process, but somewhat less intuitive than the direct interpolation with expressions and filters in AngularJS. HTML with attributes a good basis for visual editing. Small implementation. Very fast.
React JS client-side and nodejs, PHP would require port Templates are JSX (JavaScript with embedded XML) or plain JS, we would need a standalone XML format. Has the usual security benefits of DOM-based templating. 110k compressed. Very fast when used as a front-end UI library.

Implementation

[edit]

On the client-side, the library would be packaged into a ResourceLoader module targeting both desktop and mobile (but not loaded by default). On the server side, we would simply include the class files in /includes/libs/ (after a security review).

Example of client-side use

[edit]

MediaWiki will support transporting HTML blobs and in future Mustache templates. The type of template will be determined by the file extension. Code paints a thousand words

Adding templates to a ResourceLoader module

[edit]
$wgResourceModules['module.with.templates'] = array(
    'dependencies' => array(
       // provides default HTML blob transportation
       'mediawiki.templates',
       // provides mustache template support.
       'mediawiki.templates.mustache',
    ),
    'templates' => array(
        'one.html' => 'templates/example/one.html',
        'two.mustache' => 'templates/example/two.mustache',
    ),
);

Create a template

[edit]

one.html

[edit]

Example template file using HTML syntax: 

<h2>Title</h2>
<p class="desc"></p>

two.template

[edit]

Example template file using Mustache syntax: 

<h2>{{title}}</h2>
<p class="desc">{{{description}}}</p>
Shouldn't templates be precompiled? Performance will be infinities better. NRuiz (WMF) (talk) 13:47, 21 January 2014 (UTC)[reply]
ResourceLoader could compile templates and cache the result, similar to LESS "source" files for CSS. -- S Page (WMF) (talk) 19:23, 23 January 2014 (UTC)[reply]

Use them in JavaScript

[edit]
var template = mw.template.get( 'module.with.templates', 'one.html' );
var template2 = mw.template.get( 'module.with.templates', 'two.template' );
console.log( template.render( { title: 'HTML templates', description: 'Yay!' } ) );
console.log( template2.render( { title: 'HTML templates', description: '<strong>Yay!</strong>' } ) );

Prints 

<h2>Title</h2>
<p class="desc"></p>
<h2>HTML Templates</h2>
<p class="desc"><strong>Yay!</strong></p>
[edit]

Footnotes

[edit]
  1. ↑
    • Tests:
      • Test1 - Output a <div> with the same id attribute and body 100,000 times
        • MediaWiki: Html::element( 'div', array( 'id' => $vars['id'] ), $vars['body'] );
        • Twig: $twig->render('<div id="{{ id }}">{{ body }}</div>', $vars );
        • Mustache: $html = $engine->render('<div id="{{ id }}">{{ body }}</div>', $vars );
      • Test1b - Output a <div> with a different id attribute, but the same body 100,000 times
      • Test2 - Output a div for each element of a 1,000 element associative array, 1,000 times updating an element of the array on each itteration.
        • MediaWiki: foreach ( $vars['items'] as $key => $item ) { $body .= Html::element( 'div', array( 'id' => $key ), $item ); }
        • Twig: $html = $twig->render('<div id="{{ id }}">{% for key, item in items %} <div id="{{ key }}">{{ item }}</div>{% endfor %}</div>', $vars );
      • Test3 - Output a div for each element of a 1,000 element array, where each element has a different value, and update an element on each itteration.
        • MediaWiki: foreach ( $vars['items'] as $item ) { $body .= Html::element( 'p', array(), $item ); }
        • Twig: $html = $twig->render('<div id="{{ id }}">{% for item in items %}<p>{{ item }}</p>{% endfor %}</div>', $vars );
        • Mustache: $html = $engine->render('<div id="{{ id }}">{{# items }}<p>{{ . }}</p>{{/ items }}</div>', $vars );
Retrieved from "https://www.mediawiki.org/w/index.php?title=Requests_for_comment/HTML_templating_library&oldid=4690362"