Developer Portal/Implementation

The Developer Portal is a static site built using MkDocs and Material for MkDocs. It uses a custom plugin to generate PO files and integrate with translatewiki.net.

Repository: git:wikimedia/developer-portal
Staging (demo) site: developer-portal.wmcloud.org
Production site: developer.wikimedia.org
Build info: developer.wikimedia.org/build-info.txt
Analytics dashboard: piwik.wikimedia.org

How it works

Rendering page content

The purpose of the Developer Portal is to help users find key documentation pages. The site is designed to make it easy to remix documentation links into new pages tailored for different purposes. Data in the Dev Portal is organized into three main types of data:

documents: YAML files in data/documents describe key documents. Each document can be assigned to one or more categories.
categories: YAML files in data/categories describe categories. Behind the scenes, our macros/__init__.py local plugin collects categories and their corresponding documents into data that can be used in jinja templates.
markdown files: Markdown files reside in src/ subdirectories which correspond to the top-level sections of the dev portal site. Jinja template syntax inside those markdown files pulls in content from categories and documents.

Rendering the navigation

The Material theme for Mkdocs generates navigation sections based on the documents in a given subdirectory. It renders tables of contents based on the structure of an individual markdown file (its headings).

The navigation.indexes setting in mkdocs.yml implements section index pages. This causes the index.md file for a directory to be used as the landing page for a section. We use this feature because our index.md files (in /src) only include links to the other pages in the directory, so we don't need or want them to be displayed as an additional item to click in the left nav.

While it could be simpler to use a single markdown file for each site section, and rely on the TOC for navigation, this has drawbacks:

The TOC disappears on small screens, effectively removing navigation to key subsections.
Using multiple markdown files enables navigation through and between site sections via buttons at the bottom of the screen. This is a nice additional navigation option to supplement the left and/or horizontal menus.
Using a single index.md file populated via YAML for each site section means that nearly all the site structure will be rendered via YAML and templates. This means a larger number of YAML files, and a more difficult structure to understand if you're encountering the site for the first time. It's easier to understand how the site is rendered -- and thus better for future maintainers -- when more of the structure is coming from markdown and less from YAML and Jinja magic.

For more details about why we made these implementation decisions, see this Miro board (restricted access).

Search

Through the MkDocs Material theme, the Dev Portal uses Lunr for client-side, full-text search. You can see the search index at /search/search_index.json

How to use it

The Material theme for MkDocs renders content in ways that can be confusing due to our usage of templates to generate page content based on yaml docs and categories (see the /data subdirectory).

This section describes the content requirements that doc editors and creators must follow in order for the site to render correctly and consistently.

Hide TOC in subdirectory index files

Because the navigation.indexes and toc.integrate features are enabled, the table of contents is automatically hidden on index.md files in subdirectories of the /src directory. This is necessary because the content of most of these pages is navigational: the section headers in the page mirror the list of other pages in the directory/navigation section, so the navigation and the table of contents render the same content. This is only true for subdirectories of the /src directory–not for the top-level index.md nor for any other markdown files.

Mirror text copy in index files and category files

In general (but not always), each section (##) in a subdirectory index.md file should contain:

A text snippet that is the same as the category description used in the Jinja template on the subpage to which that section links
A link to another md file (subpage) in that directory

Render a document

To render a single document, add this syntax to a markdown file:

{{ document.render( "document-filename-here" ) }}

This renders the document with an h2 heading. To customize the heading level, add the level as an argument.

For example, this syntax renders a document with an h3 heading:

{{ document.render( "document-filename-here", 3 ) }}

Render a category

To render a category and its assigned documents, add this syntax to a markdown file.

Render one category per page:

{{ category.render( "category-name-here" ) }}

Render multiple categories per page:

{{ category.render( "category-name-here", 2 ) }}

Order documents in a category

To control the order that documents appear in a category, use this syntax in the document YAML definition:

categories:
    category-a: 1
    category-b: 2
    category-c: 0

In this example, the document will be ordered first in category-a, second in category-b, and unordered in category-c. Note that if a document needs to be ordered in any of its category, all listed categories must use the dictionary syntax instead of the list syntax (category-a: 1 instead of - category-a).

If a category contains both ordered and unordered documents, the unordered documents are listed first since their index is 0.

Mkdocs plugins

Macros module

The upstream macros plugin adds support for jinja templates to the markdown pages of a MkDocs website. We extend this plugin using a custom "macros" Python module. This exposes various variables and macros to the jinja layer.

Variables

categories: Dict of keyed by category name of information loaded from YAML files found in data/categories. In addition to the data found by parsing the YAML file, each value contains a "documents" list which includes every document from data/documents which was tagged with the category.
documents: Dict keyed by file name (minus extension) of document information loaded from YAML files found in data/documents.

Macros

category_data( name ): Get the data for a category or an empty dict if the category does not exist.
document_data( name ): Get the data for a document or an empty dict if the document does not exist.

JinjaWrapperPlugin

Wrap markdown files in a jinja template.

This plugin works in concert with the macros plugin to setup each markdown file as an extension of the data/includes/markdown_base.jinja template prior to it being rendered by the macros plugin. This in turn allows us to setup common imports and macros for use by all markdown files while keeping boilerplate in the markdown files themselves to a minimum.

TranslatePlugin

Tracked in Phabricator
Task T297168
Resolved

Translate content via Gnutext PO files.

This local MkDocs plugin maintains a translation catalog based on English language content extracted from the src/**/*.md Markdown documents and other configuration files as a GNU Portable Object (PO) file suitable for import by translatewiki.net. It also generates internationalized static pages based on non-English PO files found in subdirectories of data/locale/. Translation units are extracted from Markdown documents after the Jinja templates are expanded.

This plugin was heavily inspired by both mkdocs-mdpo-plugin and mkdocs-static-i18n which attempt to provide similar functionality.

The plugin hooks into a number of lifecycle events exposed by MkDocs:

on_config

Create tmp dir for generated non-English Markdown files
Scan disk for translation dictionaries (PO files)
Add discovered languages to theme's alternate language configuration
Add supported locales to search plugin config

on_files

Extract translation units from English mkdocs.yml config values
Create a parallel content tree in the tmp dir for each target language for translation

on_nav

If target language is English, extract translation units from nav config
If target language is non-English, translate non-English nav config

on_page_markdown

If target language is non-English, translate Markdown using PO data
If target language is English, extract translation units from Markdown

on_page_context

Update alternate links to point to current page in other locales

on_post_build

For each supported locale, build translated pages using a truncated version of the MkDocs internal build process
Remove obsolete translation units from English PO file
Remove generated tmp directory and contents

on_build_error

Remove generated tmp directory and contents

Local development environment

The local development environment uses Docker, docker-compose, and GNU Make to manage a container hosting the project. This container is used to build and serve the compiled mkdocs project.

Setup

Clone the repository, and start the container.

$ git clone https://gerrit.wikimedia.org/r/wikimedia/developer-portal
$ cd developer-portal
$ make start

Building the site

This command builds the site, updates the translation files, and starts a local server at http://localhost:9000/. When changing content, re-run this command to update the local server.

$ make build

Remember to build the site before submitting a patch.

Demo server and production deployment

See the docs on Wikitech for information on operating the production and demo deployments.

Analytics

The Dev Portal uses Matomo to collect pageview statistics. To view the dashboard, log in to idp.wikimedia.org with your Wikimedia Developer Account, and use the placeholder credentials to complete the second log-in step on Matomo itself.

Baseline metrics

Analytics data for the Developer Portal starts in March 2023. The monthly data below is based on the time period for which we (currently, as of Sept 19, 2024) have a complete month of data: April 2023 - August 2024:

Monthly	Pageviews	Unique Pageviews	Outlinks	Bounce rate	Actions per visit
Average	71186	60673	11777		2
Median	70234	59283	12084	60%	2.2
Min	59353	50675	8347	57%	2
Max	83300	71721	14301	66%	2.4

Pages with the most outlinks

These pages of the developer portal have the highest number of links clicked to exit the portal:

Site page (limited to top 10)	Sum of Unique Clicked Outlinks 2023 April - 2024 August
developer.wikimedia.org/get-help/	134
developer.wikimedia.org/contribute/by-language/	107
developer.wikimedia.org/get-started/wikimedia-tech/	103
developer.wikimedia.org/build-tools/apis/	85
developer.wikimedia.org/use-content/content/	71
developer.wikimedia.org/get-started/new-dev/	69
developer.wikimedia.org/use-content/featured-apps/	67
developer.wikimedia.org/use-content/tutorials/	67
developer.wikimedia.org/get-started/	63
developer.wikimedia.org/use-content/data/	60

Troubleshooting

AttributeError: 'NoneType' object has no attribute 'items'

Check for a Markdown file with an empty link. (You can search the /src directory for ]().) In the Dev Portal, links in Markdown must have a value for the URL.

Yes

[link text](placeholder)

No

[link text]()

Check for a document file with an empty categories list. If a document has no categories, the categories line should be removed.

Yes

---
title: My title
description: My description
links:
  - url: https://...
    label: Learn more
categories:
  - example-category

---
title: My title
description: My description
links:
  - url: https://...
    label: Learn more

No

--
title: My title
description: My description
links:
  - url: https://...
    label: Learn more
categories:
# no categories

TypeError: unhashable type: 'dict'

Check YAML syntax for files in which you're attempting to Order documents in a category. If you're adding an order argument to the category field, you must not prefix the category name with a hyphen:

Yes

---
title: Hackathons
description: A hackathon brings together people with different skills to work together on a common technical goal.
links:
  - url: https://www.mediawiki.org/wiki/Special:MyLanguage/Hackathons
    label: Read more on mediawiki.org
categories:
  events: 1

No

---
title: Hackathons
description: A hackathon brings together people with different skills to work together on a common technical goal.
links:
  - url: https://www.mediawiki.org/wiki/Special:MyLanguage/Hackathons
    label: Read more on mediawiki.org
categories:
  - events: 1 # hyphen in this line must be removed

How it works

Rendering page content

Rendering the navigation

Search

How to use it

Hide TOC in subdirectory index files

Mirror text copy in index files and category files

Render a document

Render a category

Order documents in a category

Mkdocs plugins

Macros module

Variables

Macros

JinjaWrapperPlugin

TranslatePlugin

Local development environment

Setup

Building the site

Demo server and production deployment

Analytics

Baseline metrics

Pages with the most outlinks

Troubleshooting

AttributeError: 'NoneType' object has no attribute 'items'

TypeError: unhashable type: 'dict'

See also