User:DBrant (WMF)/Goodies/Push notifications

Here's how push notifications work:

Client-side setup

The Android app is integrated with Firebase, and the iOS app is integrated with APNS (Apple Push Notification Service). By "integrated" we mean that the actual unique package name of the app is registered under our account in the Firebase and APNS systems.
- See the "PushNotifications" project in the Firebase console, and observe that the list of apps registered under it include all four flavors of the Android app (production, beta, alpha, dev).
- Make sure all necessary team members have access to the respective APNS and Firebase consoles (look under "Users and Permissions").
When the app is first launched on a user's device, Firebase or APNS assigns a "token" to the app, which the app receives through a callback, which the app saves permanently.
If the user logs in to their Wikipedia account in the app, we will also register the above token with the Echo extension by calling the echopushsubscriptions API. If Firebase/APNS ever sends a new token to the app, the new token must be sent to the API again. The Echo extension supports up to 10 simultaneous tokens per user. If you register more than 10, the oldest token(s) will be dropped.

Server-side setup

When the Echo extension has a notification for a user that is push-worthy, it sends a message to the push-notifications REST service.
- NOTE: We don't actually send the contents of notifications when pushing. We send a fixed message (namely checkEchoV1), which signals to the client that they should query the API for new notifications.
- The message consists of the device token(s) (registered by the client above) and the constant identifier checkEchoV1, so that recipients will distinguish it from other potential Firebase/Apple messages unrelated to Echo.
The push-notifications service is a Node.js service that runs on our Kubernetes cluster, and provides REST endpoints /v1/events/fcm and /v1/events/apns (for Firebase and Apple respectively) for receiving the push messages from Echo. (These endpoints are not for external use, and are just for Echo to dispatch messages.)
The push-notifications service is then responsible for sending the messages to the actual Firebase and Apple systems. The service is configured with private secrets (API keys) for communicating with Firebase and APNS, and uses Node libraries from these respective providers for interfacing. These libraries should be kept up to date, to stay on top of any deprecations or updates to best practices.
The push-notifications service is also responsible for one other thing:
- If the Firebase or Apple backend rejects a particular device token (because it's expired, etc), then the service will log on to Meta Wiki with a special bot account, and invalidate the bad device token. This bot account has the special user right called manage-all-push-subscriptions, which makes it able to manage (i.e. delete) other users' device tokens for this purpose.
- The bot account is called meta:User:PushSubscriptionManager2, and its password can be retrieved from our shared 1Password vault. (The password is set up as another private environment variable in the service deployment.) There is also a corresponding PushSubscriptionManager2 account on the Beta cluster for testing purposes.
When the push-notifications service receives messages from Echo, it does not forward them immediately to Firebase and/or Apple. Instead, it queues up the messages so they are sent as a batch, every 10 minutes or when the queue becomes full, whichever is first. This was done for security reasons, so that outgoing messages couldn't be correlated with actions on a specific wiki account.

Running locally

Clone the push-notifications repo.
Run npm install.
Set up environment variable(s):

For Firebase, obtain a private key json file from the Firebase console, and:

export GOOGLE_APPLICATION_CREDENTIALS="<path to API key json file>"

Edit config.dev.yaml and plug in the username/password of the bot user that manager push tokens (mw_subscription_manager_username / password). (Naturally, do NOT commit this change to the repo.)
Run npm start!

After making any changes to the code of the service, rebuild it with npm run build, and restart it again.

Deploying updates

For deploying changes and updates made to the push-notifications service, refer to the Wikitech page on Kubernetes deployments. But the gist is as follows:

When a patch is merged into the repo of the service, one of the CI steps will produce a new Docker image tag in our Docker registry. Look for a CI comment from PipelineBot that contains an "IMAGE:" and "TAGS:" line with a Docker tag, and copy that tag. It should be something like "YYYY-MM-DD-HHMMSS-publish". (example patch)
Clone the deployment-charts repo, and edit the file helmfile.d/services/push-notifications/values.yaml. In this file, update the main_app/version to be the new version number copied from above. (example patch)
Merge the patch to the deployment-charts repo, and wait a couple of minutes for the repo to be updated on the deployment server. (This cron job runs every minute.)
Log on to the deployment server, and execute the appropriate helmfile commands:

First, deploy to the staging cluster:

helmfile -e staging -i apply --context 5

If necessary, perform any testing you like on the staging server:

curl -X POST -H "Content-Type: application/json" https://staging.svc.eqiad.wmnet:4104/v1/message/fcm -d "{\"messageType\":\"checkEchoV1\",\"deviceTokens\":[\"<token1>\",\"token2\",\"token2\"]}"

And finally deploy to the production clusters:

helmfile -e codfw -i apply --context 5
helmfile -e eqiad -i apply --context 5

Client-side setup

Server-side setup

Running locally

Deploying updates

Other resources