User:Legoktm/Blog/Shellbox

In July 2020, Wikimedia received a security report that it was possible to gain remote code execution through LilyPond, a tool used to render musical scores on wiki pages (see an example) and quickly disabled the feature as mitigation. Over the next year we attempted to fix our firejail-based sandboxing system, before ultimately designing and implementing Shellbox, a sandboxing solution that uses Kubernetes and Linux containers for isolation.

A bit of background

MediaWiki has always relied on external commands to make certain functionality to work. Especially in our media handling stack, image thumbnailing is done by using tools like imagemagick, librsvg and vips, videos are transcoded using ffmpeg and so on. We also have some less visible features that use external commands, like syntax highlighting using Pygments, musical scores that use LilyPond and timelines use a homegrown Perl wrapper around ploticus.

The security of these commands has been a large issue in our overall security. Many of these commands were never written with adversarial user input in mind and have been full of remote code execution (RCE) attacks [EXAMPLES]. In 2016 we started wrapping calls to these commands with firejail, a SUID sandbox program using Linux namespaces (T135111). It was always an awkward fit as firejail was really designed and intended to sandbox desktop programs like Firefox, Chromium, VLC, etc.

In 2017 this layer of using firejail was moved into MediaWiki itself (T173370). Rather than needing to create a custom wrapper for each command we shell out to, MediaWiki would automatically wrap each command with the appropriate firejail options, allowing the caller to specify whether the command should have network access, use the execve syscall and some more. As restrictions were added to each command, we removed the manual firejail wrappers.