Translation of app string resources

Current localization documentation is available

Translations of string resources are done at translatewiki.net (aka TWN).

Repo Configuration

See Wikipedia iOS and Wikipedia Android groups in translatewiki.net Gerrit project.

Requesting Access

All of the following steps must be performed on translatewiki.net, and therefore require that you have access to the machines. To request access:

File a Phabricator ticket to request access and add it to the translatewiki.net project (see example)
1. Be sure to include the public key of the RSA key you want to use to access the machine
2. Ensure your local SSH config uses the correct identity file for the translatewiki.net host
3. The i18n team will then review your request for access, and if granted, update the translatewiki.net manifest to add your key
File another Phabricator ticket to request access to review changes (see example)
1. Create a user on translatewiki.net
2. Mention user name in Phab ticket

SSH Config

In step 2 above, your SSH config should look something like this:

Host translatewiki.net
  User bgerstle
  IdentityFile ~/.ssh/translate_wiki_rsa

TWN sync for Android

About once a week we sync string resources between our code and TWN. The terms import and export are used based on the TWN perspective. We first import new strings (English + qqq) from our code bases to TWN then export translated strings out of TWN to our code bases. The following section document how this is done at a low level.

Before starting the TWN sync make sure you have joined the #mediawiki-i18n IRC channel.

Import from Android code to TWN

ssh translatewiki.net
cd ~/wikipedia-android
git fetch origin
git checkout origin/master
repoupdate wikipedia-android
repo update wikipedia-android
php /www/translatewiki.net/docroot/w/extensions/Translate/scripts/processMessageChanges.php --group=out-wikimedia-mobile-wikipedia-android-\* --name=$USER

If the last command say "No changes" then continue with the export steps.

If it says something like "Process changes at https://translatewiki.net/wiki/Special:ManageMessageGroups" then go to that URL in your browser, review the to be imported strings. See also #Notes about flags.

Run job queue if there were changes to expedite the process (otherwise you have to wait a few minutes to make sure the import, which runs in the background, is complete:

sudo -u betawiki php /www/translatewiki.net/docroot/w/maintenance/runJobs.php

Export from TWN to Android code

This will push a new commit to Gerrit in the respective app repo(s):

repoexport wikipedia-android
repocommit wikipedia-android
exit

Go to Gerrit and run the translation tests (time ./gradlew connectedAlphaReleaseAndroidTest -Pandroid.testInstrumentationRunnerArguments.class=org.wikipedia.test.TranslationTests), then merge.

Semi-automation

#!/usr/bin/env bash
set -eou pipefail

cd ~/wikipedia-android
git fetch origin
git checkout origin/master
repoupdate wikipedia-android
repo update wikipedia-android
php /www/translatewiki.net/docroot/w/extensions/Translate/scripts/processMessageChanges.php --group=out-wikimedia-mobile-wikipedia-android-\* --name=$USER
read -p '<enter> to continue, <ctrl-c> to abort: '
sudo -u betawiki php /www/translatewiki.net/docroot/w/maintenance/runJobs.php
repoexport wikipedia-android
repocommit wikipedia-android

TWN sync for iOS

Import from iOS code to TWN

ssh translatewiki.net
cd $HOME/wikipedia-ios
git fetch --prune origin
git checkout develop
git reset --hard remotes/origin/develop
# don't ask, just do them both
repoupdate wikipedia-ios
repo update wikipedia-ios
php /www/translatewiki.net/docroot/w/extensions/Translate/scripts/processMessageChanges.php --group=out-wikimedia-mobile-wikipedia-ios --name=$USER

If the last command say "No changes" the continue with the export steps.

If it says something like "Process changes at https://translatewiki.net/wiki/Special:ManageMessageGroups" then go to that URL in your browser, review the to be imported strings.

Run job queue if there were changes to expedite the process (otherwise you have to wait a few minutes to make sure the import, which runs in the background, is complete:

sudo -u betawiki php /www/translatewiki.net/docroot/w/maintenance/runJobs.php

Export from TWN to iOS code

The steps are as follows:

# Check out new, timestamped branch to apply changes to (which will be pushed to remote at the end)
git checkout -b twn/$(date +%m-%d-%Y)

# Update QQQ with any missing strings
scripts/update-qqq.php

# Export changes from TWN and apply to current repo
# First argument is the project name, second is the repo where the project was cloned
repoexport wikipedia-ios /home/$USER

# Run git status here to make sure you have a bunch of unstaged changes to localized files
git status

# Double check no merge conflicts slipped in.
find -iname \*.strings|xargs -rd\\n grep -E '[<>]{7}'

# Fix any reversed dollar signs from RTL translations.
# TODO: this doesn't appear to work for RTL languages. Find an alternative.
find -iname \*.strings |xargs -rd\\n sed -ri 's%(\d+)\$%$\1%g'

# Check for HTML comments.
find -iname \*.strings|xargs -rd\\n grep '<!--'

# Add any new untracked localization files
git add --all

# Use script to generate standard TWN commit message
# Arguments passed in the same manner as repoexport above
repocommit wikipedia-ios /home/$USER

# Push the current branch to a remote ref w/ the same name
# This is the step where you enter your GitHub username and password, using a personal access token if necessary (see Pushing TWN Changes to GitHub below)
git push origin HEAD:twn/$(date +%m-%d-%Y)

Pushing TWN changes to GitHub

If your GitHub account uses two-factor authentication (2FA) you'll need to create a personal access token, which will be used as your GitHub password in the git push phase. If it doesn't use 2FA, it should!

Integrating new localizations

Sometimes new languages will be added for an existing localized resource. Manually verify that all localized assets are integrated into the Xcode project by checking for added files and ensuring their corresponding language is checked in the Xcode file inspector panel. TODO: automatically integrate "lproj" assets into Xcode project via xcodeproj Ruby gem.

Semi-automation

#!/usr/bin/env bash
set -eou pipefail

cd $HOME/wikipedia-ios
git fetch --prune origin
git checkout develop
git reset --hard remotes/origin/develop
# don't ask, just do them both
repoupdate wikipedia-ios
repo update wikipedia-ios
php /www/translatewiki.net/docroot/w/extensions/Translate/scripts/processMessageChanges.php --group=out-wikimedia-mobile-wikipedia-ios --name=$USER
read -p '<enter> to continue, <ctrl-c> to abort: '
sudo -u betawiki php /www/translatewiki.net/docroot/w/maintenance/runJobs.php
declare br="twn/$(date +%m-%d-%Y)"
git checkout -b "$br"
repoexport wikipedia-ios /home/$USER || :
scripts/update-qqq.php
! find -iname \*.strings|xargs -rd\\n grep -E '[<>]{7}'
! find -iname \*.strings|xargs -rd\\n grep '<!--'
find -iname \*.strings|xargs -rd\\n sed -ri 's%([0-9]+)\$%$\1%g; s/%([0-9]+)/$\1/g; s/\$d/%d/g; s/%\$([0-9]+)/$\1/g'
git add -A .
repocommit wikipedia-ios /home/$USER
echo git push origin "HEAD:$br"

Notes about flags

Note about the ignore flag: We tend to ignore translated string resource changes, unless there is a good reason for them to be done in our code and it should be reflected in TWN. (Usually translated strings come from TWN, and not the other way around.)

Note about the fuzzy flag: If the meaning of the English string has changed then mark it as fuzzy so it shows up as Outdated in the TWN interface. Then translators know they should translate the string again.

One time TWN sync setup

Set GERRIT_USER to the proper value. This is currently written for both Android and iOS but will need to be split up in the future.

GERRIT_USER=
ssh translatewiki.net
cd /resources/$GERRIT_USER/
cp ~/REPOCONF .
repo create wikipedia-android
repo create wikipedia-ios
cd wikipedia-android
git remote add gerrit ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/apps/android/wikipedia.git
git remote add origin ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/apps/android/wikipedia.git
cd ../wikipedia-ios/
git remote add gerrit ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/apps/ios/wikipedia.git
git remote add origin ssh://$GERRIT_USER@gerrit.wikimedia.org:29418/apps/ios/wikipedia.git

Follow steps in Gerrit settings to set up a new set of ssh keys so you don't need to do ssh agent forwarding.

Fixing translations on TWN site

To get to the string in TWN fast you can:

go to the Message group statistics for Android or iOS,
click on the language you want,
switch from Untranslated to Translated or All,
then enter the translated string in the Filter.
From there (in the "List" tab) you can click on Edit.

Change translations

You can change the string here but that should be left to the translators. What you can do though is to add the string !!FUZZY!! in front of the current text to mark it fuzzy/outdated. You can also look at the translation history of that particular string by clicking on the little down arrow.

Example URLs

More about Message group statistics

A given language is only exported from TWN if it has reached or exceeded a certain translation completeness threshold. Currently the threshold is set to 35% [1].

Advanced

Translatewiki.net on Gerrit

https://gerrit.wikimedia.org/r/#/admin/projects/translatewiki

For Android TWN sync see groups/Wikimedia/WikimediaMobile-android.yaml, esp. the codeMap section.

Repo Configuration

Requesting Access

SSH Config

TWN sync for Android

Import from Android code to TWN

Export from TWN to Android code

Semi-automation

TWN sync for iOS

Import from iOS code to TWN

Export from TWN to iOS code

Pushing TWN changes to GitHub

Integrating new localizations

Semi-automation

Notes about flags

One time TWN sync setup

Fixing translations on TWN site

Change translations

Example URLs

Spanish

Simplified Chinese: For zh/strings.xml use zh-hans

Traditional Chinese: For zh-rTW/strings.xml use zh-hant

More about Message group statistics

Advanced

Translatewiki.net on Gerrit