Wikimedia Labs/Per-project SaltStack remote execution

We'd like to use SaltStack for remote execution. Like everything else in labs, we want this to be per-project. Here's some possible implementations for doing so:

Using peer relationships

• Add a puppet variable, that's completely ignored by puppet called: "salt_peer_master"
• Use external pillars, doing something like the following pseudo code:

peer_masters = ldap.search('puppetvar=salt_peer_master=true')
for peer_master as peer_master:
  peer_configuration[peer_master.project]['peer_master'] = peer_master
  # of course, assume multiple peer masters
instances = ldap.search('puppetvar=instanceproject=%s' % (','.join(projects)))
for instance in instances:
  peer_configuration[instance.project]['instances'] = instance
  # of course, assume multiple instances

• Use a jinja template for salt's master configuration file. Use the "peer_configuration" hash to set the peer relationships.

Here's an example configuration:

peer:
  i-000002.pmtpa.wmflabs:
    - .*:
      minions: i-000000.pmtpa.wmflabs, i-000001.pmtpa.wmflabs

Problems with this solution

The peer relationships don't allow you to specify which peers are allowed to run which commands on which peers. It only allows you to specify which commands a peer can run on all minions. We'd need to modify salt for this. There's an approved feature request for this.

Use a simple peer relationship a wrapper script and sudo policies

We could set the following peer relationship:

peer:
  bastion.wmflabs.org:
    - .*
  bastion-restricted.wmflabs.org:
    - .*

Then, we could write a wrapper script:

salt-project <projectname> <command>

We'd need to set a grain on the instances based on the project, and have the script make calls based on the grain.

Problems with this solution

With this solution, if the bastion host was compromised, it would mean immediate compromise of all of the instances. Also, it's not easy to make this script fine grained (run this command only on this instance).