User:DKinzler (WMF)/RESTbase development setup
Daniel’s field nodes on running RESTbase for development. YMMV.
Installation:
[edit]See https://www.mediawiki.org/wiki/RESTBase/Installation
Daniel’s config: https://phabricator.wikimedia.org/P49564
In config.yaml, adjust the domain part of the path specs to match any domain (around line 45 and 67):
paths:
/{domain}/{api:v1}
Under x-sub-request-filters and for apiUriTemplate as well as parsoidHost change the domain to be the address of mediawiki as seen from inside the container. That’s probably the host’s IP. Try host.docker.internal:
/action:
x-modules:
- path: sys/action.js
options:
# XXX Check API URL!
apiUriTemplate: "http://host.docker.internal:8888/w/api.php"
# XXX Check the base RESTBase URI
baseUriTemplate:"<nowiki>{{'http://{domain}:7231/{domain}/v1'}}</nowiki>"
/page_save:
x-modules:
- path: sys/page_save.js
/parsoid:
x-modules:
- path: sys/parsoid.js
options:
parsoidHost: <nowiki>http://host.docker.internal:8888/w/rest.php</nowiki>
response_cache_control: 's-maxage=0, max-age=86400'
To only have one worker to deal with (useful for attaching a debugger):
# Number of worker processes to spawn.
# Set to 0 to run everything in a single process without clustering.
num_workers: 0
# Number of milliseconds to wait for a heartbeat from worker before killing
# and restarting a worker. 'false' means disabling the heartbeat feature.
worker_heartbeat_timeout: false
I also changed this in this logging section at the bottom, for sanity:
logging:
name: restbase
level: warn
Run in plain node container
[edit]docker run --rm -v `pwd`:/usr/src/app -w /usr/src/app -p 7231:7231 -p 9229:9229 -p 9230:9230 --name restbase node:16 node --inspect=0.0.0.0:9229 server.js
NOTE: we tell the node debugger to listen on port 9229, but we also publish port 9230. That’s because restbase uses ServiceRunner, which (depending on setup) starts worker processes for restbase, which will then use the next ports, starting at 9230.
Setting num_workers: 0 will make everything run in a single process, so RESTbase will use debug port 9229.
The output should look something like this:
~/code/restbase> docker run --rm -v `pwd`:/usr/src/app -w /usr/src/app -p 7231:7231 -p 9229:9229 -p 9230:9230 --name restbase node:16 node --inspect=0.0.0.0:9229 server.js
Should show something like:
Debugger listening on ws://0.0.0.0:9229/748a4d7e-24de-49c6-b8c6-d30a0dd8b4eb
For help, see: https://nodejs.org/en/docs/inspector
{"name":"restbase","hostname":"208c55b610fe","pid":1,"level":50,"levelPath":"error/metrics","msg":"No such metrics client: 'undefined'","time":"2023-07-17T09:59:12.281Z","v":0}
{"name":"restbase","hostname":"208c55b610fe","pid":1,"level":40,"levelPath":"warn/startup","msg":"listening on *:7231","time":"2023-07-17T09:59:12.780Z","v":0}Verify that RESTbase is up and running:
[edit]check that the port is exposed: docker container ls. Shows 0.0.0.0:7231->7231/tcp
curl http://localhost:7231/localhost/v1/ returns {"items":["page","transform"]}. Yay!
NOTE: The path must contain the hostname used in $wgServer! In the past, this has to also include any port number present in $wgServer, with the “:” encoded: e.g. http://localhost:7231/localhost%3A8888/v1/
Call Parsoid endpoint
[edit]curl http://localhost:7231/localhost/v1/page/html/Main_Page
^-- this should do something in restbase, even if it fails to connect to mediawiki to actually get the html.
NOTE: use --head to only see the response header
NOTE: until recently, the domain in the path had to include the port as well, as in /localhost%3A8888/.
Run Mocha Tests
[edit]If a RESTbase container is already running
> docker exec -it restbase /bin/bash
To start a container for running the tests:
docker run -it --rm -v `pwd`:/usr/src/app -w /usr/src/app -p 9229:9229 --name restbase-mocha node:16 bash
Then, inside the container, run:
./node_modules/.bin/mocha test
To ensure you are running on a clean slate (with SQLite), use export RB_SQLITE_FILE=`mktemp -t sqlite.XXXXXXXXXX`.
If you want to debug the tests, add --inspect-brk=0.0.0.0:9229.
NOTE: Mocha tests use their own config, config.yaml is ignored! Mocha tests will hit the beta cluster!
If Mocha tests for parsoid transformations are failing because they require HTTPS (T350353), you have to adjust the parsoid host URL. In config.fullstack.test.yaml, change the parsoid host URL so it uses HTTPS:
host: https://parsoid-external-ci-access.beta.wmflabs.org/w/rest.php.
If you are getting TLS errors, you may have to first run export NODE_TLS_REJECT_UNAUTHORIZED=0.
Not that RESTbase will connect to this host but it will send Host: en.wikipedia.beta.wmflabs.org in the headers. To re-create this on the command line, use curl -k -H "Host: en.wikipedia.beta.wmflabs.org" https://parsoid-external-ci-access.beta.wmflabs.org/w/rest.php/.
WebStorm Setup
[edit]In WebStorm, create a new run configuration of the type “Attach to Node.js/Chrome”.
Set the port to the appropriate number (9229 or 9230, depending on setup).
Set up path mappings to match the volume mount point in the docker container, e.g. map the project directory to /usr/src/app.
Using Blubber
[edit]In .pipeline/blubber.yaml:
add this for the development and test variants:
runs:
insecurely: true
Reason: sqlite needs to be able to write to ./db.sqlite3
To enable debugging, modify the entrypoint of the development and test variants:
entrypoint: [node, "--inspect=0.0.0.0:9229", server.js]
Generate docker file, using blubber:
[edit]blubber .pipeline/blubber.yaml development | docker build --tag parsoid-dev --file - .
Run RESTbase from the image blubber defined:
[edit]docker run --rm --expose 7231 --expose 9229 -p 7231:7231 -p 9229:9229 parsoid-dev
NOTE: looks like the dockerfile generated by blubber doesn’t have an EXPOSE directive. Why?...
HOW can I expose ports in blubber?
DOH: wasted an hour because I put the parameters after the image name, instead of before…
WebStorm setup
[edit]In WebStorm, create a new run configuration of the type “Attach to Node.js/Chrome” [TBD]