Jump to content

Requests for comment/Unified Zero design

From mediawiki.org
Request for comment (RFC)
Unified Zero design
Component Wikipedia Zero
Creation date
Author(s) Yurik, ABaso(WMF)
Document status implemented

In production (patch)

Background

[edit]

In order to significantly reduce varnish fragmentation and reduce the complexity, Zero team would like to unify HTML served to all Zero partners's users. The banner will be replaced by the <script> tag to include a dynamically generated, carrier-specific document.write() banner, or for the no-script browsers, a dynamically created GIF image.

Previous discussion and options were discussed at Requests for comment/Zero Architecture and Requests for comment/Data-driven Zero Varnish Configuration.

Problem

[edit]

Wikipedia Zero webpages are served to users on mobile devices with participating mobile carriers. The number of Wikipedia Zero cached pages, in excess of non-Wikipedia Zero mobile-formatted pages, is roughly: 

cached_pages = 0
foreach carrier c:
  cached_pages += c.one_or_two_subdomains_from_m_or_zero_subdomains * c.num_languages_supported

Carriers support Zero-rating of <language>.zero.wikipedia.org, <language>.m.wikipedia.org, or both. They also support up to ten customized free languages, otherwise they support all languages.

The amplification of cached pages means more hits at the origin servers than wanted, meaning slower loading pages for Wikipedia Zero users. Furthermore, the current page caching scheme employed via Wikipedia Zero introduces a challenge to differentiating such Wikipedia Zero cached pages from non-Wikipedia Zero cached pages - a problem when the Wikipedia Zero team wants to purge the cache to modify aspects of the Wikipedia Zero experience without impacting other aspects of the Wikipedia mobile-formatted experience.

[edit]

All HTML to the Zero partners will contain this at the top: 

<script src="/w/index.php?title=Special:ZeroRatedMobileAccess&zcmd=js-banner" />
<noscript>
  <img src="/w/index.php?title=Special:ZeroRatedMobileAccess&zcmd=img-banner" />
</noscript>

No-script banners

  • NoScript banner is rendered as a small, non-customizable image for the specific carrier in format "free from {{CarrierName}}". The <img> tag will not set height or width (the CSS properties width and max-width may be set to allow constraints to be automatically applied, though).
  • For MDOT, if the banner should not be shown for the specific request (e.g. this language is not whitelisted), returns 1px x 1px commons:File:Blank.gif.
  • ZERODOT request on partner network:
    • banner response is RED WARNING if lang.zero not supported; the article content still comes back, though
    • otherwise, it's the normal banner
  • ZERODOT request on non-partner network shows UNCACHED error with the IP address

JavaScript banners

  • API will return a javascript code snippet which will render partner-specific banner if required:
document.write("...");
ZRMA = {
  "enabled":true,
  "id":"234-56",
  "showImages":true,
  ...
};

<div style="background:#F4A83D;color:#735005;" class="mw-mf-banner" id="zero-rated-banner">
  <button type="submit" title="dismiss this notification" style="background:#F4A83D;" class="notify-close">
   <span style="background:#F4A83D;border-color:#735005;color:#735005;" class="notify-close-x">x</span>
  </button>
  <a href="/wiki/Special:ZeroRatedMobileAccess?showbanner=1">
    <span style="color:#735005;" id="zero-rated-banner-text" class="mw-mf-message">
      Free {{SITENAME}} from {{CarrierName}}
    </span>
  </a>
</div>
  • The <span> banner tag be wrapped in a generic <a> such that a tap/click/press anywhere on the banner brings the user to the interceptor for the banner tap action on non-JS devices. For JS-supporting devices, the "x" should remain its own distinct element in the innerHTML setter by the returned <script> tag so that when tapped it will it will guarantee banner (and "x") non-display for the next 24 hours.
  • Currently, the "x" for non-JavaScript browsers does nothing, as it's a JavaScripty thing. In the future state the "x" for the <noscript> version should be removed altogether, as it complicates matters too much.

Varnish logic

[edit]

For all mobile traffic (both ZERODOT & MDOT), set X-CS2 if from a carrier network (already implemented):

  • if HTTPS, use top value of X-Forwarded-For
  • If ip matches ANY proxy, use top value of X-Forwarded-For
  • If ip matches ANY carrier, set req.http.X-CS2 = carrier ID
if (req.http.X-CS2) {
  // TBD: Beware of legitimately formed links happening to have :ZeroRatedMobileAccess
  if (req.url ~ "(action=zeroconfig|:ZeroRatedMobileAccess)($|&|\\?)" ) {
    set req.http.X-CS = req.http.X-CS2; // Backend response will have much shorter cache lifetime
  } else {
    set req.http.X-CS = "ON"; // ON is an unreserved carrier ID, hence we use it here
  }
} // for the rest of m. or zero. traffic, do not set X-CS (but it will still vary on it)
  • In vcl_deliver, append req.http.X-CS2 to resp.http.X-Analytics
  • All traffic will continue to vary on X-CS, X-SUBDOMAIN, language, and the URL's path

Note that we are redefining the meaning of the X-CS - it used to mean carrier's ID for all valid zero traffic, or unset if coming from a non-zero source, or if the carrier did not explicitly whitelist it. With this change, X-CS will always be set if the traffic is coming from the zero carrier. On the other hand, the value of X-CS will be set to ID only for the calls to the zeroconfig API and to the Zero special page. For all other requests, X-CS will be a boolean on/off flag ("ON" or unset), thus greatly reducing cache fragmentation.

TBD

[edit]
  • image library i18n exotic character support - need a proper ttf font, same as timeline ext
  • Image width/height JS/CSS and old HTML support - image will move the page. Looks good with JS
  • Acceptable stats deviations from reality
  • If on ZERODOT okay to show article content, BUT have red banner. This is still relatively low bandwidth and should be a rare occurrence.
  • Any impacts on existing app APIs? Don't think so off top of head, but need to check

JSON Config Structure

[edit]
{
    "comment": "Usually the carrier's name", // optional
    "name": {
        "en": "Carrier"
    },
    "banner": {
        "en": "Free {{SITENAME}} from $1"
    },
    "bannerUrl": "http://...",  // Optional
    "langNameOverrides": {...}, // Optional
    "admins": [ "User" ],       // empty list by default
    "background": "#000000",    // Optional
    "foreground": "#FFFFFF",    // Optional
    "fontSize": '0.8em',        // Optional
    "showLangs": ["en", "fr"],
    "showZeroPage": false,      // Default depends on count of "showLangs": false for 1, true for 2+
    "ipsets": {   // allows to define sets of IPs
        "default": [
          "50.185.139.129",
          "54.200.159.9",
          "67.244.17.232",
          "2600:1010:b004:468b:44df:1bad:265d:21b4",
          "2601:9:7b80:92c:4d44:a9bd:2775:6375",
          "2601:9:7b80:92c:8919:44ab:cfb3:6b8c"
        ],
        "waps": [ // TODO if needed: handle when WAP and DIRECT use the same IPs,
                  // but WAP can be detected some other way, e.g. extra headers
          "71.183.236.23",
          "72.14.179.17",
          "74.71.10.241",
          "162.243.129.61",
        ]
    },
    "configs": [
        {   // this will get ipsets=["default"]
            "enableHttps": true,
        },
        {
            "comment": "This is the WAP configuration",
            "ipsets": ["waps"],      // ["default"] by default
            "enabled": false,        // true by default
            "enableHttps": false,    // false by default
            "proxies": ["OPERA"],    // ["DIRECT"] by default
            "whitelistedLangs": ["en","fr"], // all languages by default
            "sites": ["m.wikipedia"],// default=["m.wikipedia","zero.wikipedia"]
            "bannerWarning": true,   // false by default
            "disableApps": true,     // false by default
            "imageWarning": true,    // NOT implemented. false by default
        }
    ]
}

Analytics

[edit]

Current Zero analytics is based on presence/absence of the X-CS ID in the X-Analytics response header. Since we are switching from the accurate tagging of valid Zero traffic to tagging all of the carrier's traffic, X-Analytics will contain an X-CS value more frequently. This would include m.wikipedia.org and zero.wikipedia.org traffic for all languages, arriving via direct, proxied (to date, Opera and Nokia), and HTTPS. If the carrier is not zero-rating any of it, we would be overcounting.

To prevent this, we could leave all of the complex Varnish per-carrier logic intact so that X-Analytics header remains accurate. Varnish will be cleaned up once analytics is capable of further filtering (MapReduce routines?). This will give us a smooth migration path to the new design without impacting statistics numbers. This also means that in case zero team updates configuration but forgets to make corresponding changes in Varnish, statistics might be incorrect while the site's functionality will be correct.

Analytics API

[edit]

To simplify how analytics post-process traffic, Zero provides an API to present historical configurations without IPs in a concise manner.

There could be multiple configurations per carrier ID. For example, carrier could have a WAP gateway that has a separate set of external IPs, which would only support a subset of functionality - like HTTP only, or only specific languages. To accommodate that, in addition to X-CS (carrier's ID), Varnish will append another, optional value in the X-Analytics: zerosfx=a, where 'a' is the suffix to be appended to the X-CS in order to look it up in the following API response. When producing graphs, we might want separate graphs per suffix, but mostly we would want the summary graphs where all suffixes are combined.

Lastly, please note that it is possible for the X-Analytics to contain X-CS, while the API to not have a valid configuration for that time frame. If that's the case, analytics should ignore that X-CS and treat it as a regular, non-zero traffic. 

{
  '250-10a': [ 
    {
      'from':   '2010-01-01 11:22:33', // This config applies on or after this timestamp
      'before': '2013-02-01 23:44:12', // This config applies if before this timestamp
      'https': true,             // supports https traffic (true/false)
      'languages': ['en','fr'],  // if the array is empty, all languages are free
      'sites': ['m.wikipedia'],  // list of whitelisted sites. If empty, all sites (including non WP)
      'via': ['direct','opera'], // at least one value - how the traffic arrived
    },
    ... // more configuration sets with the different, non-overlapping 'from'..'before'
  ],
  ... // other zero + zerosfx values
}
Retrieved from "https://www.mediawiki.org/w/index.php?title=Requests_for_comment/Unified_Zero_design&oldid=3834461"