Parsoid/Packaging

This page is used to develop possible solutions for bug 53723.

We would like to

• build Debian packages
• still allows operations to review root-level code
• support current deployment methods (git-deploy)
• minimize manual dependency tracking and contrib updates

Deploy repo with code as submodule

Status: Decided to go with this option in the team meeting December 11, 2013. Implemented in parsoid and parsoid/deploy repos.

Principle: node will search the path for node_modules, so we can also place it in a parent directory.

Deploy repo layout:

node_modules

dependencies, updated with npm install

src

parsoid js code as submodule

debian

Debianization and upstart / systemd config submodule. Repo controlled by ops. Configs can also be directly referenced by puppet.

package.json

symlink to parsoid/package.json

localsettings.js

MediaWiki action API configuration, loaded with --config switch to the action API

With this model multiple independent deploy branches (architecture, node version etc) are possible.

Alternative option considered: node_modules as submodule

• Use submodules to manage the debian/, puppet/ and contrib/ files. Operations has merge rights on debian/ and puppet/ so that root-level files can still be reviewed.
• Automatically build debs for third parties from the full repository, either using launchpad or a labs instance.
• Continue to use git-deploy to push out the full repository including contrib/, but deploy the config files using puppet. Submodules are currently disabled for Parsoid, but can be re-enabled when the external cpp submodules are removed.

See also / random notes

• Simple packaging of the source and the upstart / systemd script should be fairly straightforward to debianize no matter what we use. No building needed, 'just' tar up the repository and install the config.
• npm modules can be converted to basic debs with https://npmjs.org/package/npm2debian
  • does not work so well in my testing though Gabriel Wicke (GWicke) (talk) 23:23, 16 December 2013 (UTC)
• https://github.com/wikimedia-incubator/debianize aims to simplify debian packaging too, but focuses on automake/autoconf. Unclear if this might be useful to us.
• Simple deb packages can be converted to rpms with alien. If we include node_modules, then we should only depend on node and (optionally) logrotate. Node seems to be called 'nodejs' in most distros: [1]
• http://debdelta.debian.net/ looks like a bandwidth-efficient way to update Debian packages
• mini-dinstall is an alternative to reprepo, which we are currently using for the wmf repos. Reprepo only keeps the very last version of each package, which makes downgrades.. difficult.
• List of node.js / JS packages in Debian

Deployment

[13:14] <Ryan_Lane> well, we can deploy the code itself using trebuchet
[13:14] <Ryan_Lane> puppet can install the upstart and dependencies
[13:15] <Ryan_Lane> you don't need to understand salt to use trebuchet ;)
[13:15] <Ryan_Lane> https://wikitech.wikimedia.org/wiki/Trebuchet#Adding_a_new_repo
[13:16] <Ryan_Lane> also look at how parsoid is configured in puppet: manifests/role/deployment.pp
[13:17] <Ryan_Lane> and note that parsoid has a really shitty init script and that instead you can just use service.restart <service> in the checkout_module_calls config

Building debs for deployment

If we wanted to use debs for deployment, we could do the following:

• use a debian repo manager that keeps old versions (mini-dinstall for example)
• set up a deb builder script that can be run from deployers with sudo, somewhat along these lines:

git clone <repourl>
cd repo
git submodule update --init
rm -r debian # throw away unvetted debian dir
ln -s <opsrepo>/parsoid/debian .  # link in ops-vetted debian dir
# add a changes entry with a new version number (automatically incremented)
dpkg-buildpackage ...
# upload to the repo and return the new version

Then, to deploy: apt-get install parsoid=<version>, using dsh or salt.

Ideally we'd also deploy to a few canary systems first and (automatically) see that things are alright before upgrading the remaining systems.