User:DWalden (WMF)/Thumbor migration

Project documentation

For Phabricator board, team members involved, etc. see 2022_Thumbor_updates.

The Phabricator board is: https://phabricator.wikimedia.org/tag/thumbor_migration/

Feature documentation

Thumbor allows users to see resized versions of media hosted on a wiki (e.g. Commons). For example, for thumbnails.

It can be used for any media including images, videos, PDFs, etc.

Test documentation

Where to test it

Thumbor is available on the beta and production environments. However, to test the latest version you need to install it locally.

How to install locally

git clone https://gerrit.wikimedia.org/r/operations/software/thumbor-plugins/
cd thumbor-plugins

Create a thumbor.conf in the same directory. Here is the one I currently use phab:P37284.

If you want to run it in docker (recommended):

blubber .pipeline/blubber.yaml dev > Dockerfile
make build

Otherwise, if you want to run it on "bare metal":

python3 -m pip install -r requirements.txt
python3 setup.py build
python3 setup.py install --user
thumbor --port 8800 --conf=thumbor.conf -a wikimedia_thumbor.app.App

Thumbor should then be running on port 8800.

To check it is running, visit: http://localhost:8800/thumbor/unsafe/320x/https://upload.wikimedia.beta.wmflabs.org/wikipedia/en/1/1b/CuteGoat.jpg

(See also README.)

Coverage criteria

Types of media (PNG, JPG, OGV, webm, 3D, PDF, DjVU, SVG, etc.)

Oracles

Previous versions of thumbor (e.g. those installed on beta or production)

Risks

Quality criteria

Techniques

Blink testing
Automatic comparison (e.g. using imagemagick's compare or GIMP)

Test data

Any media (including video, pdf) on Commons can be passed to Thumbor. There are lots of places to find images.

https://commons.wikimedia.beta.wmflabs.org/wiki/Special:NewFiles (allows you to filter by types of media, e.g. vector, bitmap, video)
https://commons.wikimedia.beta.wmflabs.org/wiki/Special:MIMESearch (allows you to filter by mime type)
allimages API (see script)

Test tools and scripts

Script to compare an instance of thumbor running locally with thumbor on beta: https://gitlab.wikimedia.org/dwalden/thumbor-testing/-/blob/main/comparison_prototype.sh
- You should change line 31 with output from the below script
Script to find lots of media for testing: https://gitlab.wikimedia.org/dwalden/thumbor-testing/-/blob/main/allimages.py

Logs and debugging

To see more detailed logs on the command line, run with the -d -l debug flags, like:

thumbor -d -l debug --port 8800 --conf=thumbor.conf -a wikimedia_thumbor.app.App

Current testing status

Test strategy

Compare the thumbnails generated by the latest version of thumbor with the older version running on beta.