Continuous integration/Understanding build failures

This is adapted from a post by Ashar Voultoiz on wikitech-l

---

Whenever a unit test breaks, mw-jenkinsbot will spam the #mediawiki ^connect channel.

Example:

[22:20:36] <mw-jenkinsbot>	Project MediaWiki-phpunit build #119: FAILURE in 2 min 4 sec: https://integration.wikimedia.org/ci/job/MediaWiki-phpunit/119/

The bot does not report since when the tests are broken.

To fix them you have to get to the Jenkins Dashboard and find the most recent successful build, the build after that is ran on the revision that introduced the breakage.

On the build page (such as MediaWiki-build-119) the failures are listed under "Test result".

To view the test running output, go to Console Output from the sidebar.

     [exec] There was 1 failure:
     [exec] 
     [exec] 1) StoreBatchTest::teststore
     [exec] counts wrong 0 0
     [exec] Failed asserting that <integer:1> matches expected <integer:0>.
     [exec] 
     [exec] /var/lib/jenkins/jobs/MediaWiki-phpunit/workspace/mw-core/tests/phpunit/includes/filerepo/StoreBatchTest.php:71
     [exec] /var/lib/jenkins/jobs/MediaWiki-phpunit/workspace/mw-core/tests/phpunit/includes/filerepo/StoreBatchTest.php:91
     [exec] /var/lib/jenkins/jobs/MediaWiki-phpunit/workspace/mw-core/tests/phpunit/MediaWikiTestCase.php:64
     [exec] /var/lib/jenkins/jobs/MediaWiki-phpunit/workspace/mw-core/tests/phpunit/MediaWikiPHPUnitCommand.php:31
     [exec] /var/lib/jenkins/jobs/MediaWiki-phpunit/workspace/mw-core/tests/phpunit/phpunit.php:60
     [exec] 

If you have found the build in which a test started to break, go to Changes from the sidebar, there you will find the exact svn revision that broke the tests. Alternatively, you can use svn blame to find out the revision that broken the test.

So suppose it was introduced by r89191 .. Check it out and run tests :

  $ svn co -r 89191
   <snip>
  $ php parserTests.php --quiet --filter 'bug 93'
   This is MediaWiki version 1.19alpha (r89191).
   <snip>
   Passed 0 of 1 tests (0%)... 1 tests failed!
  $

That one was broken on commit anyway: r89191

The BlockTest were broken too, most probably due to its rewrite. You can easily reproduce the test suite and most project manager we love the easy report functionality (--testdox):

  $ cd tests/phpunit
  $ ./phpunit.php --filter BlockTest --testdox
   PHPUnit 3.5.13 by Sebastian Bergmann.

   ApiBlock
    [ ] Make normal block

   Block
    [ ] Initializer functions return correct block
    [x] Bug 26425 block timestamp defaults to time
    [ ] Bug 29116 load with empty ip
    [ ] Bug 29116 new from target with empty ip
  $

Have fun :-)