Wikibase/Programmer's guide to Wikibase

This page describes the workflow for developing features for Wikibase.

Whenever you develop something — develop a complete new feature or just make a small fix — you create a new branch. Whenever it makes sense, you squash the changes into one commit, rebase it on the current master, and submit that commit for review on Gerrit.

Once the changes are on gerrit, they need to be reviewed and approved for merging by another developer.

So, to keep things moving, you should review at least as many changes as you submit for review!

Requirements

Wikibase/Installation
You need to have git-review installed (see Gerrit/Workflow#git-review for instructions).

Understanding Dependencies

Modifying a stand alone component

Get a git clone of the component if not already done so. This clone can be put anywhere on your machine. No web server or database setup work required.
Install composer if not already done so.
Run composer update in the components root directory.
Makes your changes. The tests can be run by executing phpunit in the root directory. An alternative PHPUnit runner can be used, and pointed to the phpunit.xml.dist configuration file in the root directory.
Submit your changes for review.

Modifying a component that depends on MediaWiki

Set up a working MediaWiki installation if you have not already done so.
Get a git clone of the component if not already done so. Put it in the extensions dir of your MediaWiki installation.
Include the component's entry point in LocalSettings.php if it's not already present. If you also have any of its dependencies in the extensions directory, make sure they are not loaded.
Run composer update in the component's root directory.
Make your changes.
Submit your changes for review.

Modifying components together

In some cases one wants to make changes in component A, and then verify component B, which depends on A, still works correctly with the new version of A. Two approaches can be taken here. The example scenario consists out of doing development on Wikibase and sometimes one of its components. You have a Wikibase install which got its dependencies via composer. Now you want to make a change to WikibaseDataModel, one of its dependencies, and see if Wikibase still works.

Getting everything from source

composer update --prefer-source

Composer will get everything from source. This means one can modify one of the obtained packages, commit the change, and push it for review. The git remote "origin" will be set and work out of the box for GitHub projects. In case of projects hosted by gerrit, git review needs to be initialized first.

If the packages where already obtained without specifying the prefer-source flag, delete the "vendor" directory and run the update command with the flag.

When using this approach it is important to realize the version of WikibaseDataModel diverges from what is defined in the Wikibase package (in composer.json). One should thus not run a composer update before having completed the task, as this might override the changes you made.

Using distinct projects

One can get a clone of each component one wants to contribute to and do a composer install in it. This has the advantage one does not go behind composers back as in "Getting everything from source". Another advantage is that when opening the directory with an IDE, it will only see the source of the package and its dependencies.

1. Make changes to WikibaseDataModel, test them for WikibaseDataModel, and push them to a branch on a git server that you can access

2. In your Wikibase project, update composer.json to know about the git server (using the repositories section).

"repositories": [
	{
		"type": "vcs",
		"url": "https://gerrit.wikimedia.org/r/mediawiki/extensions/WikibaseDataModel.git"
	}
],

The url can be a local directory containing a git repo, ie

"repositories": [
	{
		"type": "vcs",
		"url": "/home/j/workspace/WikibaseDataModel"
	}
],

3. Specify the branch you want to get code from

"require": {
        // ...
        "wikibase/data-model": "dev-branchName",
        // ...
}

4. Run "composer update"

Troubleshooting

If you install Wikibase and get an error such as:

Fatal error: Class 'Wikibase\TemplateRegistry' not found in /srv/mediawiki/extensions/Wikibase/lib/WikibaseLib.php on line 97

You need to run composer install.

Contribute to the code of Wikibase

Follow the workflow established documented on Gerrit/Advanced usage#Submitting patches.
For a smallish change (basically, anything that you would develop by yourself on a single day), use the following workflow.

Preparing your Patch

Work on that branch until your feature is ready for review:

git status # Make sure you are on "master" and create the branch from there

# On branch master

git checkout -b feature_branch_xyzzy # Create your feature branch locally

# Now write some code. See the Git commands "add", "rm" and "mv" to add, remove or rename files.

# add unit tests for your code.

# Run the Wikibase test group. It should pass all unit tests.

cd tests/phpunit

php phpunit.php --group Wikibase

When the code is tested and you're ready - push it for review.

git rebase origin/master # rebase to the latest master, and resolve any conflicts

git commit --all # In the Gerrit world you should do this only once per branch! Remember to follow the commit message guidelines.
git commit --all --amend # If you have already committed to the branch and want to change something. Make sure to leave the "Change-Id" line intact.

Caveat: commit --all will automatically include all modified files, but not new files!

git review # send the change to gerrit for review

You need to have git-review installed for this (see Gerrit/Advanced usage#Reviewing code for instructions). And for every repository you want to use git-review with, run git review -s once, to set it up for that repository.

Fixing your Patch

See Gerrit/Advanced usage#Working on an existing change set

Collaboration and Dependent Changes

See Gerrit/Advanced usage#Create a dependency

Amending Changes Other Changes Depend On

Say you have two changes (let's call them A and B, with the commit hashes A1 and B1) where B depends on A, as described above (so, A1 is B1's parent commit). Now A needs to be amended for some reason, creating A2. In this situation, B becomes obsolete, because B1 depends on the obsolete commit A1. To fix this, do the following:

download change B: git review -d B, where B is B's change ID.
rebase B1 on A2, creating B2: git rebase A2, where A2 is A2's commit hash.
1. resolve any conflicts, use git rebase --continue to continue the rebase.
Hint: if git rebase tells you that there are no changes after you resolved some conflicts, use git rebase --skip to resolve this.
use git log to verify that the last commit in the log is still B, and the commit message still contains B's change ID.

Now the situation is resolved as far as git is concerned, but we still have to tell gerrit. So now we submit B2 to gerrit for review: git review -R

Gerrit should now figure out what you have done, showing a new change set B2 for B, and mark B as no longer obsolete, but still depending on A.

If you have more changes (say, C, D, and E) that depend on B, you have to repeat the procedure rebaseing C on B, D on C, and so on.

Workflow for Reviewing Changes

There are two parts to reviewing a change: approval and verification. Approval is conceptual and formal agreement, while verification is about technical compliance.

Approval

To approve a change, visit it's page on gerrit and look through the diffs (note the convenient "next" and "previous" links on the diff pages). Try to understand the intend and implementation of the change, and check the coding style.

The most important part of course is that the new code is sensible conceptually and that it is implemented correctly. But there are some more things that should be considered:

is the code internationalized? Are system messages used where they should be?
does the code follow the code conventions with respect to naming, indentation, spacing, etc?
is the code thoroughly documented on the class and method level?
does the code come with the appropriate unit (and/or integration) tests?
Is the code efficient, does it make use of caching where appropriate?

If you have questions or find anything objectionable, click inside the diff to leave an inline comment. Note that your inline comments will be saved as drafts but not directly published - this also applied to replies to comments! You need to actually post a review verdict (below) to publish your comments.

It's also useful to actually try the code you are reviewing. To do that, first download the change into a local branch:

 git review -d <number>

...where <number> is the (decimal) change number from the gerrit URL or the (hex) change id given on the gerrit page. This will create a local branch for the change set.

Now, try the new functionality (or whatever the change provides) at least superficially.

After you are finished your review, you should post your verdict by pressing the review button. Select the appropriate level of approval (between -2 and +2), add a comment if you like, and hit "Publish Comments". If you just want to publish your inline comments (or replies), leave the rating at 0.

Unless you have also done verification, leave the verification section untouched.

If you think the change can be merged, proceed to the merging section.

Verification

Verification makes sure that the change is technically complied, that is, it

applies cleanly to the latest master
passes all tests it provides
causes no other tests to fail

This verification would ideally done automatically by Jenkins (see our request on Bugzilla). For the moment however, we are stuck doing this manually. So, here are the steps:

First, download the change into a local branch

 git review -d <number>

...where <number> is the (decimal) change number from the gerrit URL or the (hex) change id given on the gerrit page. This will create a local branch for the change set.

Next, try to merge the latest version of the master branch into the change's branch by doing:

 git pull origin master

Note: technically, we want to check whether the change can be merged into master, not vice versa. But since 3-way-merge is a commutative operation, the resulting source code will be the same either way.

If there are conflicts, the change should be rejected (set to -1) as broken.

Testing

There are three different frameworks

Selenium tests (mw:Selenium Framework, old page)
QUnit tests (mw:Manual:JavaScript unit testing)
PHPunit tests (mw:Manual:PHP unit testing)

Also note the page at mw:Requests for comment/Unit testing.

Browser Testing for Wikidata

Prerequisites

Install Ruby on your local machine
- For Linux, the easiest way is to use RVM, as described on https://rvm.io
```
\curl -L https://get.rvm.io | bash -s stable --ruby
```
  Alternatively you can install Ruby by running
```
sudo apt-get install ruby2.1
```
- For Windows, use the installer from https://rubyinstaller.org/downloads.
  - Also, download the Development Kit from https://rubyinstaller.org/downloads, extract it, cd into it and run
```
ruby dk.rb init
```
```
ruby dk.rb install
```
- On MacOS Ruby usually is included. You can also use homebrew to easily get Ruby by running
```
brew install ruby
```
- Further helpful information can be found on https://www.ruby-lang.org/en/downloads/

Update RubyGems
- ```
gem update --system
```

Install Bundler
- ```
gem install bundler
```

Install required packages

git clone https://gerrit.wikimedia.org/r/mediawiki/extensions/Wikibase

```
cd tests/browser
```
```
bundle install
```
Make sure you have the latest version of Firefox installed

Setup test configuration

You can specify your configuration in two ways. You can either set the environment variables yourself or by setting them in tests/browser/environments.yml.
Mandatory settings are:

mediawiki_url
mediawiki_user
mediawiki_password
item_namespace
property_namespace
item_id_prefix
property_id_prefix
language_code
browser

Have a look at the "custom:" section in tests/browser/environments.yml. If you want to use the settings from there you have to specify it in your environment:

export MEDIAWIKI_ENVIRONMENT=custom

Please also note that mediawiki_url must end with a slash (i.e. /)!

Running tests

Switch into the tests directory:

cd tests/browser

Run all tests:

bundle exec cucumber

You can also just run a specific feature by specifying the feature-file.
e.g. run the label tests:

bundle exec cucumber features/label.feature

You can also just run a specific scenario of a feature by specifying the line-number of that scenario inside the feature-file.
e.g. run test which checks the label UI elements:

bundle exec cucumber features/label.feature:14

You can also run scenarios with a specific tag.
e.g. run tests which are tagged with @ui_only:

bundle exec cucumber features/label.feature --tag @ui_only

Some scenarios need a valid login given in the environment variables WB_REPO_USERNAME and WB_REPO_PASSWORD:

mediawiki_user=user
mediawiki_password=password

Using a specific browser

By default your browser tests will use Firefox. If you want them to run in a different browser you can specify that by setting an environment variable manually or in tests/browser/environments.yml

Linux/Unix:

export BROWSER=chrome

Windows:

set BROWSER=chrome

For browsers other than Firefox you would need to install an appropriate driver, e.g. chromedriver for Chrome. See the following links for details:

Run the tests in the cloud

For testing different OS/Browser combinations it makes sense to run the tests on an external service. Saucelabs offers this possibility and you'll just need a few commands to run your tests in their cloud.

You'll have to set your Saucelabs username and accesskey manually in your environment or in tests/browser/environments.yml:

Linux/Unix:

export SAUCE_ONDEMAND_USERNAME=myusername
export SAUCE_ONDEMAND_ACCESS_KEY=myaccesskey

Windows:

set SAUCE_ONDEMAND_USERNAME=myusername
set SAUCE_ONDEMAND_ACCESS_KEY=myaccesskey

Make sure you point to a public accessable Wikibase instance in your config/config.yml! Do NOT use the live site for testing!
Now you can start your tests as described in the examples above and watch them running in your Saucelabs account.

Run tests in parallel

You can run features in parallel with the parallel_test gem. If you run bundle update the gem will be already installed.

You can then run your features in parallel by doing:

bundle exec parallel_cucumber features/

By default it will run as many features in parallel as your machine has CPUs. You can specify the number with -n. This will run e.g. 3 features in parallel:

bundle exec parallel_cucumber features/ -n 3

If you want to pass options to underlying cucumber, you can pass them as string with -o. This will run the tests that have the @smoke tag and will skip tests that have the @skip tag:

bundle exec parallel_cucumber features/ -o '--tags @smoke --tags ~@skip'

Run tests headless

If you're on a Linux machine you can run your tests on a headless browser. That makes them slightly faster and you wont have annoying browser windows popping up.

The only prerequisite is that you have Xvfb installed:

apt-get install xvfb

If you want to run your tests headless, all you have to do, is setting the appropriate environment variable or add the setting in tests/browser/environments.yml:

export HEADLESS=true

And then run your tests as usual.

QUnit tests

Set $wgEnableJavaScriptTest = true in LocalSettings.php to enable use of Special:JavaScriptTest.

To run the QUnit tests, simply point your browser to the following URL:

http://localhost/w/index.php/Special:JavaScriptTest

To exclusively run Wikibase's own unit tests (skipping core and other extensions), apply a filter:

http://localhost/w/index.php/Special:JavaScriptTest?filter=wikibase

PHPUnit tests

To be able to run tests I had to set absolute paths in LocalSettings.php, this could perhaps be better solved in suite.xml

Run PHPUnit tests

The merge was successful, next try running the Wikibase unit tests:

 cd tests/phpunit
 php phpunit.php --group Wikibase

This is just for finding any obvious breakage early on. If this passes, try running all unit tests:

   php phpunit.php

If any tests fail, the change should be rejected (set to -1) as broken.

To get rid of failing tests outside our groups, for example all kinds of failures from testing with SqLite which probably are not common here, try --exclude-group sqlite

   php phpunit.php --exclude-group sqlite

MediaWiki provides a custom wrapper for the standard phpunit command, which is located in tests/phpunit/phpunit.php. It supports all command line options and parameters the original phpunit command does (plus a few arcane ones we don't need).

To run all tests, including tests for any extension you have configured in LocalSettings.php:

 cd tests/phpunit
 php phpunit.php

To run only tests for the Wikibase group (provided you have the Wikibase extensions configured in your LocalSettings.php):

 php phpunit.php --group Wikibase

To run one specific test class:

 php phpunit.php ../../extensions/Wikibase/repo/tests/phpunit/includes/api/ApiGetItemId.php

To run one specific test function:

 php phpunit.php --filter testMyThing

(note that ---filter actually takes a regular expression).

Advanced Options and Configuration

In order to catch more problems when running tests, enable strict error reporting in your LocalSettings.php:

 error_reporting( E_ALL | E_STRICT );
 ini_set( 'display_errors', 1 );
 
 $wgShowExceptionDetails = true;
 $wgShowSQLErrors = true;
 $wgDebugDumpSql  = true;
 $wgShowDBErrorBacktrace = true;

There are several things you may want to change in test/phpunit/suite.xml:

Turning off verbose output, removing all the annoying details about skipped and incomplete tests:

 verbose="false"

Disabling test timeouts (and other strict checks):

 strict="false"

Or increasing the timeout:

 timeoutForSmallTests="8"

Note: make sure never to check in your modified suite.xml! you could also place your modified suit.xml in a different location and tell phpunit where to find it:

 php phpunit.php --configuration /path/to/phpunit/suite.xml

For testing different setups (e.g. different database engines), you can tell phpunit to load a different LocalSettings.php file:

 php phpunit.php --conf /path/to/server/LocalSettings.php

Writing PHPUnit Tests

Test case classes should use the following code skeleton:

/**
 * ...
 *
 * @ingroup Wikibase
 * @ingroup Test
 *
 * @group Wikibase
 * @group Stuff
 *
 * ...
 */
class MyStuffTest extends MediaWikiTestCase { 
    function testFoo() {
        ...
    }
}

The important bits are:

extend MediaWikiTestCase
use a class name (and matching file name) that ends in "Test".
use method names that start with "test" for actual test functions. You can have other (helper) functions that don't use that prefix.
use @ingroup in the class comment to indicate to doxygen into which group documentation about this class should go. "Wikibase" and "Test" should be there.
use @group in the class comment to indicate to phpunit to which test group this test belongs. "Wikibase" should be there.

There are some special groups that trigger special behavior if you apply them:

@group Database causes phpunit to set up temporary database tables for use by the test, so that modifications performed by the test are not visible in the actual wiki database. This must be done for all tests that need the database, because Jenkins will run tests without @group Database without a valid database connection.
@group medium causes phpunit to consider the test to be "medium heavy" instead of the default "small". This will apply a greater timeout to the test when phpunit runs in strict mode.

Registering Extension Test Cases

Update the registerUnitTests method in the Wikibase.hooks.php file (resp. WikibaseLib.hooks.php or WikibaseClient.hooks.php) with file paths to the individual test files. This is all that is necessary to get it up and running. To test this from the command line use

php phpunit.php --group Wikibase

When everything is in place on a central test server it should be possible to run tests for the Wikidata extension just like any other tests.

File structure

Test cases should be placed in the tests/phpunit directory under the extension's directory. Below that, follow the directory structure used by the php files under test: if the file to test is in includes/api, put the test case in tests/phpunit/includes/api, and so on.

Post your verdict

To post your verdict, press the review button on the change's page on gerrit. Select the appropriate level of approval (-1 if it failed or +1 if all is well), add a comment if you like, and hit "Publish Comments". Leave the approval rating as it is, unless you also went through the approval process.

If you think the change can be merged, proceed to the merging section.

Merging

Once the change is approved (+2) and verified (+1), it can be merged in to the main line's master branch. To do so, click the submit change button on the change's gerrit page. Gerrit will then post a message containing the result of the merge attempt. Either the change gets merged, or the merge failed for some reason.

If the merge failed, please use the Review button to change the verified level to -1 (broken).

xDebug

xdebug is a php module that allows debugging of php code directly in the IDE. With the help of the xdebug module, you can debug web requests with break points, variable inspection, etc.

Installation

Ubuntu

Under Ubuntu, you can just install the xdebug package:

 apt-get install php5-xdebug

Mac OS X

(assuming that you have homebrew installed):

 brew tap josegonzalez/homebrew-php # set up the centralized repository for PHP-related brews by josegonzalez
 brew install php53-xdebug

Resources

http://xdebug.org/wizard.php xdebug setup wizard

PHP Configuration

Fox your IDE to be able to talk to xdebug, you need to enable remote debugging. Put this into your configuration:

xdebug.remote_enable=1

This can be done in your php.ini or a related configuration file. On Ubuntu, the correct file would be:

 /etc/php5/apache2/conf.d/xdebug.ini

Browser Setup

In order to debug a web application with xdebug, you need to start a debug session somehow. The simplest way to do this is a bookmarklet in your browser.

Firefox

To start an xdebug session for some site (typically localhost, but could be anything), create a bookmarklet with the following content:

javascript:(/**%20@version%200.5.1%20*/function()%20{document.cookie='XDEBUG_SESSION='+'PHPSTORM'+';path=/;';})()

You can also use the bookmarklet generator: http://www.jetbrains.com/phpstorm/marklets/

If you are using PHPStorm, you may also want to install the JetBrains Firefox extension to enable JavaScript debugging. Note however that this plugin actually listens on a TCP port - make sure it's not open to the outside...

Chrome

TBD

IDE Setup

Setup with PHPStorm

Set a Breakpoint
Start listening on the debug interface (click the red phone/handset button; it should turn green).
Use your browser to request a page
If necessary, configure path mappings for your project (PHPStorm will ask you when you hit a breakpoint for the first time).
You should now be able to step through the code line by line, inspect variables, etc.

Very useful: detailed walk-through with screen shots

Setup with Eclipse

http://bogdan-albei.blogspot.de/2010/06/php-remote-debugging-with-xdebug-and.html

TBD

Setup with NetBeans

TBD

Resources

Michael Hunter's You Are Not Done Yet (pdf), a comprehensive checklist of what can and should be checked. It seems to be aimed at Windows based desktop applications, but many points still apply to web applications.