Requests for comment/PHP Virtual REST Service

Request for comment (RFC)
PHP Virtual REST Service
Component	Services
Creation date	12 December 2013
Author(s)	Aaron Schulz, Gabriel Wicke
Document status	implemented approved at 2014 architecture summit , first implementation in phab:T1218

Problem description

Our current PHP bindings for the use of REST-style (web) services are suffering from several issues:

Per-service issues like authentication need to be dealt with manually for each request.
Multiple requests can't easily be performed in parallel, which prevents us from leveraging parallelism to speed up request processing.
There is no transparent and generic load balancing and fail-over support. This can also be worked around with extra infrastructure (separate load balancers, automatic failover setups), but this comes at the price of extra latency and complexity.
Transparently changing service implementations to use a local class, other network protocols or a mock server is not well supported. Combined with the common use of global variables for configuration variables like host names this makes unit testing very difficult. Back-end optimizations often require changes to all code using the service.

With more web services in the backend infrastructure it might be worth spending some time on raising the level of abstraction in our interaction with REST services from PHP.

Goals

Convenient to use: Handle repetitive low-level detail in backend handlers.
Don't obfuscate REST interfaces: Expose REST style interfaces faithfully so that developers don't need to learn several names for the same thing.
Provide a Virtual REST Service that abstracts over protocols and backends: Use a high-level interface that captures the interaction with the service interfaces, but don't hard-code how those services are accessed or implemented (compare: VFS).
Be open to the world: Also support use as a simple CURL client, but encourage the use of more convenient and abstracted backends for ease of use, performance and test-ability.

API sketch

This is a very early straw-man API design that maps fairly easily onto a curl_multi based implementation:

// $vrs is the VirtualRESTService object that is dependency injected
// In this example, two web services are mounted:
//   /math/v1/: a global math service
//   /wiki/v1/: storage service for this wiki (enwiki in this example)
//              including revisioned page content at /wiki/v1/pages/

// Add a new entry
$res = $vrs->run( 'PUT', '/math/v1/96d719730559f4399cf1ddc2ba973bbd.png', 
                       array( 'body' => $value ) );
// Also set some headers explicitly. These are stored along with the value, and returned for web requests.
$res = $vrs->run( 'PUT', '/math/v1/96d719730559f4399cf1ddc2ba973bbd.png', array( 
                        'headers' => array( 'Content-type' => 'image/png' ),
                        'body' => $value,
                    ) );

// Add several entries in a batch, returns array of results
$res = $vrs->run( 
    array(
            array( 'PUT', '/math/v1/96d719730559f4399cf1ddc2ba973bbd.png', array( 'body' => $value1 ) ),
            array( 'PUT', '/math/v1/96d719730559f4399cf1ddc2ba973bbd.png', array( 'body' => $value2 ) ),
    )
);

// Read one entry back using convenience shortcut method equivalent to 
// $store->run( array ( array( 'GET', '/math-png/96d719730559f4399cf1ddc2ba973bbd.png' ) ) );
$res = $vrs->GET( '/math/v1/96d719730559f4399cf1ddc2ba973bbd.png' );

// Run a batch, returns array of results
$res = $vrs->run( 
    array(
        // Get the HTML of the current [[Main Page]]
        array( 'GET', '/wiki/v1/pages/Main_Page?rev/latest/html' ),
        // Also read some math image (don't ask why)
        array( 'GET', '/math/v1/96d719730559f4399cf1ddc2ba973bbd.png' ),
        // And increment a counter
        array( 'POST', '/wiki/v1/clicks/Main_Page/button1', array( 'form' => array( 'increment' => 1 ) ) )
    )
);

// Read a view counter and increment a click counter in one go
$res = $vrs->run( 
    array(
        array( 'GET', '/wiki/v1/pageviews/Foo' ),
        array( 'POST', '/wiki/v1/clicks/Foo/button1', array( 'form' => array( 'increment' => 1 ) ) )
    )
);

Backend handlers

Similar to VFS the VirtualRESTService can mount storage backend handlers at paths. Backends can be HTTP-based, but can also be transparently mapped to other protocols or local PHP code. Local code could use FauxRequest to call the PHP API synchronously, mock a storage service for testing, or provide a light-weight sqlite storage service for small wikis.

In the case of backends that map to protocols curl supports, the handler needs to 1) convert a simple path request to a full curl request, and 2) handle responses coming back from curl. Non-curl protocols will additionally require an integration with the curl_multi loop, which is fairly straightforward if they provide a non-blocking 'do some work' interface similar to curl_multi.

$vrs = new VirtualRESTService();

// Mount a global math storage service
$vrs->mount( '/math/v1/',
  new MathService ( array(
    'hosts' => array (
        'https://10.1.1.246/bits/v1/math-png/',
        'https://10.1.1.247/bits/v1/math-png/',
        'https://10.1.1.248/bits/v1/math-png/'
    ), /* auth etc */
  ) )
);

// Mount a revision storage service for the current wiki as /wiki/
// RESTBase could use e.g. Cassandra as storage backend
// Example: [[Main Page]] at /wiki/v1/pages/Main_Page?rev/latest/html
$vrs->mount( '/wiki/', 
  new RestbaseService( array (
    'hosts' => array(
        // Only uses HTTPS backends for now
        'https://10.1.1.146/enwiki/',
        'https://10.1.1.147/enwiki/',
        'https://10.1.1.148/enwiki/'
    ), /* auth etc */
  ) ) 
);

Using paths for dispatching lets us set up a simple and unified namespace. Backend implementations can be reused for several services by instantiating them for each service with a different prefix.

The thin wrapper over REST-style interfaces makes new features in those directly available without a need to upgrade client bindings. Bulk operations can be performed across several backends.

Tasks to be handled by backend implementations:

Outgoing request setup

authentication: retrieve / refresh auth key, add auth headers etc
URL and more generally request munging
- Select form encoding for post requests (multipart/form-data vs application/x-www-form-urlencoded)
- Accept-encoding default if not specified in request
- Add Content-MD5 for PUT requests

It should be possible to implement this by passing the full request information to each backend for general munging.

It might be worth considering future support for asynchronous auth key retrieval, even if the initial implementation can be kept simple by doing this synchronously. The key will be cached in the backend object after the first request and will likely not need refresh during a single PHP request.

Response handling

error handling: auth refresh, retry, reporting
GZIP transfer-encoding handling if not done by HTTP library backend used
- MWHttpRequest does not yet support it, but post-2011 curl can with the right option
JSON parsing for Content-Type: application/json (with request option to disable)

Related RFCs

RFC: Services and narrow interfaces -- Making the case for service-oriented architecture
RFC: Storage service -- REST storage service
RFC: Content API -- Public REST content API based on the storage service