Jump to content

VERP/GSOC Progress Report

From mediawiki.org

Long story: The BounceHandler extension got installed everywhere in the Wikimedia production cluster after https://gerrit.wikimedia.org/r/#/c/198220/. Please find more information about the extension over here: Extension:BounceHandler

GSoC Project Wrap Up report

[edit]

We ( Jeff Green, legoktm and myself ) could deploy the bouncehandler extension in the beta labs instance ( beta.wmflabs.org ) and it now generates VERPed emails ( Demo - https://en.wikipedia.beta.wmflabs.org/wiki/Special:EmailUser ) within the pencil down date. Our implementation strategy was like :

  • Deploy bouncehandler to beta, and enable only VERP generation.
  • Route VERP-bounces to bouncehandler API in beta by exim change ( https://gerrit.wikimedia.org/r/#/c/155753/ ).
  • Enable bounce handling and let the extension run in logging only mode for a month, so that no fast un-subscription happens.
  • Enable the user-email-invalidate function after observing a threshold of bounce frequency.
  • Implement the same to the production

Due to various and consistent help from various staff and volunteers from Wikimedia, other than my mentors, specially Mark, Hoo, Bryan, Nemo_bis - we could get the extension deployed safely in beta. The deployment process was really tedious, as we met with new scenarios over there, which was coped up easily.

We could write two separate extensions for MediaWiki:

  • BounceHandler: The MediaWiki extension to generate VERPed email return-paths on every send email and holds the 'bouncehandler' API that can handle an incoming email bounce. The extension is currently in beta.
  • SwiftMailer: An alternate mailer for MediaWiki.

My mentors foresaw that that project would take more time, so we started almost one month before our project was selected which helped us getting the extension deployed in time. I also thank Nemo_bis for pointing the VERP project to me few months before, and to my mentors who volunteered to help.

GSoC Project Progress Report

[edit]

Community Bonding Period

I was active in the Wikimedia community since November 2013, so I concentrated on getting more feedback on the project -- specially the shift from PHP/ Pear mailer to third party software - Swift mailer. The shift requires a lot of consience and I kept on getting feedback, through Wikitech-I and the bugzilla report. Details of the mails are attached below. I had already started building the project environment by March 2014 with help of Jeff and legoktm, so was polishing on its working -- porting it to wikitech labs ( under project name: mediawiki-verp ).

Please follow my blog at :- http://tttwrites.wordpress.com/ to watch regular updates on my GSoC Project

March 2014

[edit]
  1. Built local instance of Wikimedia Email-Web server model.(https://www.mediawiki.org/wiki/VERP/MicroTasks )
  2. Model consisted of two virtualboxes, Box1 (runing exim4 and MW core) sending emails via Box2 (running postfix).

April 2014

[edit]
  1. Modified test environment: box1 has MW running -> sends the email -> intercepted by box2 -> routed to Box2 /var/mail/root. Box2 has external connection via NAT.
  2. Box2 rejects the mail, Box1 exim produce the bounce to wiki@wikimedia.org in /var/mail/wiki

May 2014

[edit]

Week 1 and 2: May 1 to May 19

[edit]
  1. Shifted the above local instance to WikiTech labs ( under project mediawiki-verp ). box1verpnop sends the mail having box2verpnop as the smarthost which rejects all mails to wiki@wikimedia.org. The bounce is created by box1 in the /var/mail/root of box1verpnop.
  2. Discussion on shifting from PHP mailer to Swift mailer.
  3. Overcome the SSH slow response time by mosh-ing into the Wikitech Server.

Week 3 : May 19 - May 26

  1. Working on shifting the default PEAR mailer in mediawiki to Third Party - SwiftMailer.
  2. Submitted patch in gerrit (https://gerrit.wikimedia.org/r/#/c/135290/)
  3. Asked review for the shift at ([[1]])

June 2014

[edit]

Week 4 and 5 : May 27 - June 7

  1. Working on improving https://gerrit.wikimedia.org/r/#/c/135290/, now needs to pass Jenkins tests
  2. Got private repo for swiftmailer created at https://github.com/wikimedia/mediawiki-core-vendor-swiftmailer
  3. Got swiftmailer code hosted at https://github.com/wikimedia/mediawiki-core-vendor-swiftmailer/tree/5.2.0-patch by reporting bug, https://bugzilla.wikimedia.org/show_bug.cgi?id=66110
  4. Added Composer installation for Development clusters from vendore/swiftmailer, by patch https://gerrit.wikimedia.org/r/#/c/137538/
  5. On the networking side, made a return mail to a specific address handled by a pipe transport, hosted in Wikitech-labs, Project:- mediawiki-verp, machine - box1verpnop
  6. Blog posts on Forwarding mails to PHP script - http://tttwrites.wordpress.com/2014/06/07/forwarding-mails-to-a-php-script-with-exim4/
  7. Blog posts on Exim configs ( http://tttwrites.wordpress.com/category/technical/exim/)
  8. Blog posts on Composer ( http://tttwrites.wordpress.com/category/technical/composer/)
  9. Blog posts on creating release tag from an existing tag - used to prepare the SwiftMailer repo ( http://tttwrites.wordpress.com/2014/06/06/composer-loading-specific-tags-instead-of-branches/ )

Week 6 : June 9 - June 15

  1. Started implementing VERP ( https://gerrit.wikimedia.org/r/#/c/138655/ )
  2. Creating a new Extension - BounceHandler - will be uploaded into gerrit soon. Now available at ( https://github.com/tonythomas01/BounceHandler )
  3. Installed IMAP server dovecot, getting script to fetch and evaluate bounces into hard/soft bounce ( https://github.com/tonythomas01/exim4box1verp/blob/handlingBHM/bhm/script.php )
  4. Requested for new repo at: mediawiki/extensions/BounceHandler
  5. Blog post on IMAP installation and configuration (http://tttwrites.wordpress.com/2014/06/13/using-imap-with-dovecot-in-wikitech-labs-instance/)

Week 7 : June 16 - June 22

  1. Submitted various patch-sets to build up the Bounce handler extension:

July 2014

[edit]

Week 8,9 : June 23 - July 5

[edit]
  1. Got merged :-
  2. Started writing new extension - SwiftMailer to add an alternate for UserMailer
  3. Still to be merged :-

Week 10 : July 6 - 13 2014

  1. Got Merged:-
  2. Working on :-
  3. Articles I wrote on the progress:-

and more at :- http://tttwrites.wordpress.com/category/wikimedia/

Week 11-12 : July 14 - 27 2014

  1. Got Merged:-

Week 13-14 : July 28 - Aug 10 2014

  1. Got merged :-
  2. Still to get merged :-
    1. https://gerrit.wikimedia.org/r/#/c/152164/ - Added test to bouncehandler extension to check regex functions

Week 15 : Aug 11 - Aug 17 2014

  1. Got merged :-
  2. Got Merged:

Deployment highlights after GSoC period

[edit]
  1. Beta
    1. Feb 2, 2015: Beta bounce handling complete. Wikitech announcement . A separate mx, mx.beta.wmflabs.org has been setup to route beta emails.
    2. $domain part of all sent-out messages from beta set as beta.wmflabs.org
    3. Return bounces are piped to the bouncehandler API of meta.wikimedia.beta.wmflabs.org
  2. Production
    1. Feb 2, 2015 : Installed bouncehandler on group0 wikis after https://gerrit.wikimedia.org/r/#/c/186242/
    2. Mar 3, 2015: BounceHandler installed on group1 wikis after https://gerrit.wikimedia.org/r/#/c/191937/
    3. Mar 30, 2015: BounceHandler deployed 'everywhere' in production after https://gerrit.wikimedia.org/r/#/c/198220/
    4. Configurations on productions, available at : https://wikitech.wikimedia.org/wiki/BounceHandler_Configurations

Special thanks to all the reviewers, mentors and the WMF team.

Retrieved from "https://www.mediawiki.org/w/index.php?title=VERP/GSOC_Progress_Report&oldid=4556408"