Jump to content

User:Dan Shick (WMDE)/Drafts/Doodorial

From mediawiki.org

Contents


  • Part 1
    • Get started with the mwcli development environment
    • Get a development version of MediaWiki running locally
  • Part 2
    • Load an extension
    • Make a first extension
  • Part 3
    • Look at general extension mechanisms in MediaWiki
    • Add some basic first extension functionality
  • Part 4
    • Look at making some real world changes to real world extensions
  • Part 5:
    • Install the Wikibase extension and get the extension working you your local development environment
  • Part 6:
  • Part 7:
    • Walkthroughs of some example extensions
  • Part 8:
    • Mob programming, build a bulk entity creation API.




Getting set up



Docker images

While you are waiting please get a head start by downloading these docker images


docker pull defreitas/dns-proxy-server:2.19.0docker pull jwilder/nginx-proxy:0.10docker pull docker-registry.wikimedia.org/dev/stretch-php74-fpm:3.0.0docker pull docker-registry.wikimedia.org/dev/buster-apache2:2.0.0docker pull mariadb:10.8

IDE

To write some code, you’ll need an IDE.



*

Syntax highlighting
  • Auto completion
  • Code cleanup
  • Easy navigation and documentation displays
  • etc.


Use the IDE of your choice, but we recommend the 2 below linked to their instructions:



Examples in this presentation use VSCode.

Install mwcli

mwcli contains a Docker-based development environment for MediaWiki.

This is the development environment that we will be using.


We need to install this using the following guide:


Follow the guide linked below:

https://www.mediawiki.org/wiki/Cli/guide/Installation


Mw version


Using mwcli to create a development environment

Once we have mwcli installed, we can set up the development environment for the first time.

This will also:*

clone MediaWiki
  • install needed dependencies
  • install an initial development site



Follow the guide linked below:

https://www.mediawiki.org/wiki/Cli/guide/Docker-Development-Environment/First-Setup


Once doing this you should have a local MediaWiki install running in the development environment running at default.mediawiki.mwdd.localhost using the port that you chose during setup (default 8080)


File:Image4.png.png


More on mwcli

There are many parts of the development environment that we are using that we have not yet used.

You should find the help test of the cli helpful for discovering these features.


  • mw docker stop: Stops all currently running containers
  • mw docker start: Starts containers that were running before
  • mw docker destroy: Deletes all containers and associated data (such as databases)
  • mw docker mailhog create: Creates a mailhog service for receiving email
  • mw docker phpmyadmin create: Creates a phpmyadmin service for viewing sql databases



You can find the full online reference for the CLI at https://www.mediawiki.org/wiki/Cli/ref/mw_docker


Environment check

Let’s make sure that our development environment is still running.


mw docker docker-compose ps


You should see that most services show a “State” of “Up”, with one or two showing an “Exit 0”.


If they instead show as existed, then please restart the services.


mw docker resume


If the list of services is empty then you must have run `destroy` at some point and will need to reinitialize your development environment.


mw docker mediawiki create
mw docker mysql create
mw docker mediawiki install —dbtype=mysql


You should then be able to see your wiki


http://default.mediawiki.mwdd.localhost:8080/ << You may need to change the port to be your own


You can find this with


mw docker env get PORT


Writing some code

Load the BoilerPlate extension

We now have MediaWiki code on your machine in the directory that you selected earlier.


You’ll need to open your IDE so that you can see this code.


In VSCode it’ll look something like this:


File:Image2.png.png


Now let’s add the BoilerPlate extension, a mostly blank extension template that displays 'Hello, world!' like text on pages to serve as an example.

https://www.mediawiki.org/wiki/Extension:BoilerPlate


Clicking “browse repository” in the side bar will take you to the git repo

https://gerrit.wikimedia.org/g/mediawiki/extensions/BoilerPlate


File:Image7.png.png


We can then clone the repository into the mediawiki extensions directory.


cd $(mw docker mediawiki where)/extensionsgit clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/BoilerPlate"


The result of this should look something like the below screenshot:



And finally we can enable the extension within MediaWiki by altering our LocalSettings.php file to add the following line:


wfLoadExtension( 'BoilerPlate' );


At this point your LocalSettings.php should look something like this.


File:Image1.png.png


And if you navigate to the Special:Version page of MediaWiki you should see that the Boiler plate extension is loaded.

File:Image12.png.png


Success!

You are running MediaWiki in a development environment and have loaded an additional extension!

Create your own extension


General Guide: https://www.mediawiki.org/wiki/Manual:Developing_extensions


We are going to make a custom extension called “TutorialWorld”


Extracted from the guide, to start you will need to:


  1. Make a “TutorialWorld” directory in the extensions directory
  3. Set manifest_version to 2 and a type of “other”



You should end up with something that looks like this:


{"name": "TutorialWorld","author": "Adam","url": "https://www.mediawiki.org/wiki/Extension:TuitorialWorld","description": "Magic things for TutorialWorld","version": "0.1","license-name": "GPL-2.0-or-later","type": "other","manifest_version": 2}
And then load your extension in LocalSetting.php


wfLoadExtension( 'TutorialWorld' );


If you navigate back to Special:Version, you should now see your extension in the installed extensions list.


File:Image13.png.png

Inside BoilerPlate


The extension.json file for BoilerPlate specifies what the extension is, how it works, and how it connects to MediaWiki.

Full details of the individual parts can be found at https://www.mediawiki.org/wiki/Manual:Extension.json/Schema


In the case of BoilerPlate this is…


General metadata

This stuff appears on Special:Version among other places

File:Image3.png.png


Where the code is that should be loaded
MediaWIki will instruct PHP to load classes from the directory that you specify


Configuration of the extension
In BoilerPlate this exposes 2 feature toggles for the extension

File:Image2.png


Hook Handlers, and Hook usages
Hooks allow custom code to be executed when some defined event happens
BeforePageDisplay is used in BoilerPlate which: “Allows last minute changes to the output page, e.g. adding of CSS or JavaScript by extensions.”

File:Image3.png


Internationalisation files
Adds translations of messages used by the extension to the MediaWiki internationalisation system


Frontend resources
Adds UI resourced to the resource system of MediaWiki

File:Image5.png


A version of the extension registration manifest


Adding the BeforePageDisplay hook

One of the features that the BoilerPlate extension contains is the ability to add some text to the bottom of all pages (or “vandalize” them).


This can be enabled by setting the `wgBoilerPlateVandalizeEachPage` setting to true in your LocalSettings.php


$wgBoilerPlateVandalizeEachPage = true;


We can see this setting defined in the extension.json file above.


Once set, if you reload the MainPage, you should see the extension adding text to the end of the page.


At this stage it will be beneficial to open your browser developer console, and disable caching for requests. Particularly if you don’t see this text appear right away.


File:Image14.png.png


Hooks


We can see the code that makes this happen in Hooks.php


This is a namespaced class called Hooks

The full name including namespace is MediaWiki\Extension\BoilerPlate\Hooks

The class implements a Hook interface from MediaWiki called BeforePageDisplayHook.

This interface defines an interface covering the one function implemented in the class called onBeforePageDisplay which contains the functional code.


The code that runs when the hook is called:#

Checks an extension configuration variable
  1. If it is true
    • Add an oojs-ui-core module (defined by MediaWiki)
    • Add some HTML to the bottom of the page



File:Image10.png.png


If we jump back to extension.json we can see how this function connects to MediaWiki and the hook system.


  1. A HookHandler is registered with the name “BoilerPlateHooks” pointing at the Hooks class that we just looked at above.
  2. The specific hook “BeforePageDisplay” is registered pointing to the hook handler “BoilerPlateHooks”



File:Image11.png.png


So from MediaWiki to Extension the flow looks something like this:

MediaWiki code -> Named hook -> Hook Handler -> Hooks class -> Hook interface method -> Extension code

Implementing BeforePageDisplay

Now that we have looked at an example of the BeforePageDisplay hook being used.

Let’s try and implement some different text being added to every page.


We will dedicate ~ 15 minutes to individually working through this problem and asking questions.

After 15 minutes we will go through the bullet point lists with a screen share to fill in any gaps.


Throughout these instructions, try to use the BoilerPlate extension files for inspiration.


  1. Create an src directory in the TutorialWorld extension
  2. Register the AutoloadNamespaces for the extension in extension.json
  3. Create a new class implementing the BeforePageDisplayHook interface in the src directory (Use different text!)
  4. Register the class as a HookHandler in extension.json
  5. Register the Hook in extension.json



As a stretch goal, try to make the text configurable!


Your resulting page might look something like this (Note; the “goat”):


File:Image8.png.png

Alternate ways to extend MediaWiki

Hooks are one of the major ways to extend MediaWiki or other extension functionality.

There are many hooks in MediaWiki https://www.mediawiki.org/wiki/Manual:Hooks#Available_hooks

And many more hooks in extensions…


Other common extension methods would be:*

API endpoints



All of these methods follow the same rough layout as hooks:*

Read the MediaWiki documentation on the extension mechanism
  • Register the mechanism in extension.json in the appropriate place
  • Implement the appropriate code per the extension mechanism


General notes*

Some docker images don't work well for ARM cpus. (for me(dennis) everything works fine : ))
  • Docker Engine install doesn’t install docker-compose (so hello world will work, but starting with mwcli will complain)
  • Mwcli:other notes section was done together with the segment beforehand, instead of together with the pair programming segment.
  • Looking for the right hooks seems to be a skill in itself. Does any documentation for this exist?






Wikibase for local development

We aim to develop some code that extends Wikibase at the end of this workshop.


So let’s install Wikibase into our local development environment.


This will be very similar to when we fetched the BoilerPlate extension in workshop 1.


cd $(mw docker mediawiki where)/extensionsgit clone "https://gerrit.wikimedia.org/r/mediawiki/extensions/Wikibase"


TODO

git submodule update --init --recursive


Wikibase has composer dependencies that we need to also download making use of the composer merge plugin.


Create a file in the mediawiki/corei directory called `composer.local.json` with the following content.


{ "extra": { "merge-plugin": { "include": [ "extensions/*/composer.json", "skins/*/composer.json" ] } }}


This example file is included in `composer.local.json-sample`, so you can simply run the following:


cd $(mw docker mediawiki where)
cp composer.local.json-sample composer.local.json


We then need to make composer update its dependencies.


mw docker mediawiki composer update


File:Image9.png.png


Once we have the code on our system, we can go ahead and load the extension by altering our LocalSetting.php file to add the following lines.

( for now we will only load the Repository part of Wikibase. )


## Wikibase RepositorywfLoadExtension( 'WikibaseRepository', "$IP/extensions/Wikibase/extension-repo.json" );require_once "$IP/extensions/Wikibase/repo/ExampleSettings.php";


Note: Ticket relating to getting rid of the requirement for ExampleSettings.php to be loaded


Once loaded, we also need to run update.php


mw docker mediawiki exec -- php maintenance/update.php --quick


We should then be able to navigate to Special:Version on our wiki, and see wikibase installed there


File:Image5.png.png


You can go ahead and try to create a new item using Special:NewItem to ensure everything is working correctly.


Stable interface policies


There are a couple of stable interface policies that will be relevant to MediaWiki and Wikibase development.


Note: The Wikidata stable interface policy was written primarily with Wikidata.org in mind, and does not mention Wikibase, or Wikibase releases much in its current state.

It would likely make sense to iterate on this…


Key points summarized:


    • Changes of stable interfaces (such as public APIs) will be communicated
    • Things considered stable
      • JSON and RDF output
      • Wikibase Web API (api.php)
      • Linked data interface (Special:EntityData)
      • Wikidata Query Service
      • Wikibase LUA Library
      • JavaScript Hooks documented in hooks-js.md
    • A selection of things considered UNSTABLE
      • Raw Wikibase content stored by MediaWiki and returned by the MediaWiki core API
      • Wikibase PHP code
      • Wikibase JavaScript code
      • The HTML DOM structure
    • Using Code
      • It is generally stable to call public methods on a class instance.
      • It is generally not stable to construct a class (instantiate).
      • It is generally not stable to extend a class (subclass) and not stable to implement an interface.
    • There is a documented deprecation and removal process
    • There is a documented definition of an extension “ecosystem” that is recognized



Wikibase PHP hooks


There is developer documentation for Wikibase PHP hooks


Looking at the current Repository PHP hooks*

Addition of DataTypes and EntityTypes


  • GetEntityByLinkedTitleLookup
    • Allows extensions to add custom EntityByLinkedTitleLookup services.
    • Example usage:
  • Very old, should probably be removed…
    • WikibaseTextForSearchIndex
  • Unused
    • WikibaseContentModelMapping





Break…

Walkthrough: WikibaseManifest

WikibaseManifest is an extension that combines metadata about a Wikibase installation exposing it as a simple API. The goal is to help toolmakers write tools that can target any Wikibase.


Taking a look at extension.json


The extension makes use of:*

3 configuration options



The extension is made up of around 23 actual code files.


Some Wikibase services / configurations are used:*

    • In order to report via the API the RDF URIs that are used
    • In order to report the namespaces that entities exist in locally



You can see example API output at https://addshore-alpha.wiki.opencura.com/w/rest.php/wikibase-manifest/v0/manifest

Walkthrough: WikibaseCirrusSearch

This extension implements ElasticSearch-based search for default Wikibase entities (Items and Properties)


Taking a look at extension.json


The extension makes use of:*

12 configuration options
  • Code is located in the src directory and Wikibase\Search\Elastic namespace
  • 9 hooks are used (1 from Wikibase)
  • MediaWiki i18n / Localization system



Taking a look at the 1 Wikibase hook that is used.

The hook overrides some fields of entities that are registered.

The fields that are altered can be found in WikibaseSearch.entitytypes.php

Specifically:

Def::ENTITY_SEARCH_CALLBACK

Def::SEARCH_FIELD_DEFINITIONS

Def::FULLTEXT_SEARCH_CONTEXT

These fields relate to the hook used and are documented in the entity types documentation.


Ultimately this extension replaces the default SearchHelper, with a SearchHelper that will look at elastic search.

Walkthrough: AutomatedValues

Wikibase extension that allows defining rules to automatically set labels or aliases based on Statement values.


Taking a look at extension.json


The extension makes use of:*

3 configuration options
  • Code is located in the src directory and ProfessionalWiki\\AutomatedValues namespace
  • 5 hooks are used
  • MediaWiki resource loader system is used



Looking at the hook that enabled the main functionality*

onMultiContentSave
  • Get the content that is being saved
  • If it is an entity with statements
    • Apply the relevant defined rules to the content being saved



Walkthrough: WikibaseEDTF

TODO



Break…


Programming time

We will try to peer / mob program a bulk entity creation REST API.


You are more than welcome to follow through the development process locally if you want to. Alternatively focus on the screen share and participating.


Jakob will drive, and Adam will facilitate the direction with input from the whole group.


Relevant reading


Relevant documentation:*



Relevant Wikibase code points:*

EntityStore


NOTE: This is PHP code, so it is unstable per the current stable interface policy.


Planned user input


The user should be able to submit a list of entities to be created by the API in the BODY of a POST request to a `batchcreate` route.


A body could look something like this:


{ "entities":[ { "type": "item", "labels": { "en": { "language": "en", "value": "hello goat" } } }, { "type": "item", "labels": { "en": { "language": "en", "value": "potato land" } } } ]}

(Entity format is the same as that which is used via wbeditentity)


And a response could look something like this:


[ { "id": "Q1", "rev": 2 }, { "id": "Q2", "rev": 3 }]

(In the case of an error, return an error key for that entity instead)


Goals


We have a few ordered goals to structure our approach:*

Register a route
  • Implement some code for the route responding with some hardcoded JSON
  • Handle input from the user (echo it back)
  • Deserialize a list of entities submitted by the user in the body of the request, responding with how many were received
  • Create the entities the user is submitting



And a couple of stretch goals:*

Require permissions for API usage
  • Limit the number of entities that can be created in a request to 100
  • Use a factory for the registration of the handler
Retrieved from "https://www.mediawiki.org/w/index.php?title=User:Dan_Shick_(WMDE)/Drafts/Doodorial&oldid=6540218"