Take a REST with HttpBuilder-NG and Ersatz

11 September 2017 ~ groovy

This is a long post and there is a lot of code to look through. If you would rather follow along using the completed code, you can find it in its GitHub project rest-dev.

This blog post is going to be a bit more self-serving and a bit longer than my usual posts. I will be walking through the process of implementing a REST client using HttpBuilder-NG (v0.18.0) and then testing it against an Ersatz Server (v1.5.0) to mock out the endpoints.

Let’s say we work in a big company that is implementing a bunch of microservices. Our team is working on a service that will interface with a service being created by another team doing concurrent development - say they are creating an internal user management service. Our team will need to perform operations against their service before it actually exists. In discussions between the two teams, we have fleshed out a RESTful interface contract which looks something like this:

GET /users - list all users, responds with list of users
GET /users/{id} - get specific user, responds with single user
POST /users <user> - create new user, responds with created user
PUT /users/{id} <user> - update existing user, responds with updated user
DELETE /users/{id} - delete a user, 200 means success

Nothing shocking there, but now while they are developing the actual endpoints, you are developing a client. We need a way to simulate their user API in a realistic manner so we can develop with at least some level of confidence. This is one of the use cases where Ersatz Server comes in handy.

We can quickly define a mock for each of the end points and then write client code against it. First, we will need a User object. Based on our shared contract, the User looks like the following:

@Canonical
class User {
    Long id
    String username
    String email
}

Next we need to setup a Spock test which will be used to simulate the API and test our client code. A basic Spock test with an Ersatz server is shown below (if you are not familiar with Spock, I suggest reading through the docs to get a quick feel for it before moving forward):

UserClientSpec.groovy
class UserClientSpec extends Specification {

    @AutoCleanup('stop')
    private final ErsatzServer server = new ErsatzServer()

}

This code creates our ErsatzServer instance for us and registers it to be stopped after each test method.

For our REST endpoints, we will just start from the top and implement the GET /users endpoint first.

UserClientSpec.groovy
def 'retrieveAll'() {
    setup:
    List<User> users = [
        new User(100, 'abe', 'abe@example.com'),
        new User(200, 'bob', 'bob@example.com'),
        new User(300, 'chuck', 'chuck@example.com')
    ]

    server.expectations {
        get('/users').called(1).responder {
            code 200
            content users, APPLICATION_JSON
        }
    }

    UserClient client = new UserClient(server.httpUrl)

    when:
    List<User> result = client.retrieveAll()

    then:
    result.size() == 3
    result[0] == users[0]
    result[1] == users[1]
    result[2] == users[2]

    and:
    server.verify()
}

The client method for this endpoint will be named retrieveAll() so, we will use that as the test name. We setup a few users that will be returned by the call and then configure the Ersatz expectations. The expectations are defined using a DSL to describe each expected request and then to define the response that request will return. In this case we are expecting a GET request with the path /users only once, which will return a status code of 200 and the configured list of users as a string of JSON. We then use the client object (not defined yet) to make the server call and then verify that we got our list of users back and that the server expectation was actually called.

It seems like a significant chunk of code to drop all at once, but if you read though it, it’s actually pretty straightforward.

The first problem we run into when trying to run this code is that the UserClient class does not exist yet, so let’s create that next.

UserClient.groovy
class UserClient {

    private final HttpBuilder http

    UserClient(final String host) {
        http = HttpBuilder.configure {
            request.uri = host
        }
    }
}

We are using HttpBuilder-NG (the core client in this case) to make the HTTP calls. It also uses a DSL for configuration. In this case we define the base URI to be a host that we pass in - if you look back at the test we see that it’s the ErsatzServer host in that case. This will be the root of all requests. Now, to make our test happier, we need to implement the retrieveAll() method:

UserClient.groovy
List<User> retrieveAll() {
    http.get(List) {
        request.uri.path = '/users'
        response.parser(JSON) { ChainedHttpConfig config, FromServer fs ->
            json(config, fs).collect { x -> x as User }
        }
    }
}

This method will make a GET request to the /users path on the configured host. Note that we also need to configure a parser to handle the incoming response data, which is a list of User objects serialized as JSON.

Now, if we go back and run our test, we get a nasty error about parsing JSON content on the Ersatz Server side:

groovy.json.JsonException: Unable to determine the current character, it is not a string, number, array, or object

The current character read is 'r' with an int value of 114
Unable to determine the current character, it is not a string, number, array, or object
line number 1
index number 1
[restdev.User(100, abe, abe@example.com), restdev.User(200, bob, bob@example.com), restdev.User(300, chuck, chuck@example.com)]

This means we need to add an encoder to the Ersatz Server configuration so that it knows how to encode the response it is sending back - in this case it will serialize a list of User objects as JSON to be sent as the response. We can configure this on the ErsatzServer constructor as:

UserClientSpec.groovy
@AutoCleanup('stop')
private final ErsatzServer server = new ErsatzServer({
    encoder(APPLICATION_JSON, List) { input ->
        "[${input.collect { i -> toJson(i) }.join(', ')}]"
    }
})

I just used the groovy.json.JsonOutput.toJson(Object) method for simplicity. Now, when we run the test it succeeds. At this point we have implemented and tested our client against a real endpoint. I say real because Ersatz creates an instance of an embedded Undertow server and configures the expected endpoints on it. The client code is hitting a real and standard web server with all of the expected server behavior. What you do have to be careful of with this kind of testing is that the contract with the other team does not change. This mocked testing is only as good as the configured expectations and if left unmaintained could drift far from the reality of the production endpoints - something to be aware of.

But we have other endpoints to define and clients to implement. Next, we will handle the single user retrieval case, the retrieve(long) method (GET /users/{id}). Our test for this method looks very similar to the first test:

UserClientSpec.groovy
def 'retrieve'() {
    setup:
    User user = new User(42, 'somebody', 'somebody@example.com')

    server.expectations {
        get('/users/42').called(1).responder {
            code 200
            content user, APPLICATION_JSON
        }
    }

    UserClient client = new UserClient(server.httpUrl)

    when:
    User result = client.retrieve(42)

    then:
    result == user

    and:
    server.verify()
}

Notice that in this case, we are configuring only a single user in the response. Learning from our last test, we know that we will also need to configure an encoder to handle single User objects. This one is even simpler and makes our constructor look like:

UserClientSpec.groovy
@AutoCleanup('stop')
private final ErsatzServer server = new ErsatzServer({
    encoder APPLICATION_JSON, User, Encoders.json
    encoder(APPLICATION_JSON, List) { input ->
        "[${input.collect { i -> toJson(i) }.join(', ')}]"
    }
})

For the single object case we just define the default JSON encoder. Ersatz takes the stance that if you need/want encoders and decoders you need to configure them rather than having them provided out of the box. It keeps the configuration less surprising and more explicit.

The client code for the GET /users/{id} endpoint is as follows:

UserClient.groovy
User retrieve(final long userId) {
    http.get(User) {
        request.uri.path = "/users/${userId}"
    }
}

which along the same lines as our first client method, we will need to add a response parser for deserializing the incoming JSON response. We can configure shared response parsers in the main HttpBuilder.configure() method that we have in our constructor, so that they will be available to all HTTP method calls. The client constructor now looks like:

UserClient.groovy
UserClient(final String host) {
    http = HttpBuilder.configure {
        request.uri = host
        response.parser JSON, { ChainedHttpConfig config, FromServer fs ->
            json(config, fs) as User
        }
    }
}

This uses the NativeHandlers.Parsers.json method and casts it as a User object to satisfy our object typing.

When we run our tests again, we see that they are both successful. That’s enough for the GET requests, let’s move on to something different. The POST /users <user> endpoint is tests as the others are:

UserClientSpec.groovy
def 'create'() {
    setup:
    User inputUser = new User(null, 'somebody', 'somebody@example.com')
    User createdUser = new User(42, inputUser.username, inputUser.email)

    server.expectations {
        post('/users') {
            called 1
            body inputUser, APPLICATION_JSON
            responder {
                code 200
                content createdUser, APPLICATION_JSON
            }
        }
    }

    UserClient client = new UserClient(server.httpUrl)

    when:
    User result = client.create(inputUser)

    then:
    result == createdUser

    and:
    server.verify()
}

In this case we are expecting a POST method with a User as the body content, serialized as JSON. When the request is successful we respond with the user data which also includes the id. To decode the incoming request content we need to add a decoder to the ErsatzServer constructor:

UserClientSpec.groovy
@AutoCleanup('stop')
private final ErsatzServer server = new ErsatzServer({
    encoder APPLICATION_JSON, User, Encoders.json
    encoder(APPLICATION_JSON, List) { input ->
        "[${input.collect { i -> toJson(i) }.join(', ')}]"
    }

    decoder(APPLICATION_JSON) { byte[] bytes, DecodingContext dc ->
        Decoders.parseJson.apply(bytes, dc) as User
    }
})

For the most part it is just the provided JSON decoder with the result cast as a User object. Now, for our client implementation

UserClient.groovy
User create(final User user) {
    http.post(User) {
        request.uri.path = '/users'
        request.body = user
        request.contentType = JSON[0]
    }
}

We just use the post() method and configure the request body content, which we will need a means of encoding into the outbound JSON format. Our client constructor now becomes:

UserClient.groovy
UserClient(final String host) {
    http = HttpBuilder.configure {
        request.uri = host
        request.encoder JSON, NativeHandlers.Encoders.&json
        response.parser JSON, { ChainedHttpConfig config, FromServer fs ->
            json(config, fs) as User
        }
    }
}

For the encoder, we can use the one provided with the library. Run the tests again and we see that everything is green.

I am going to skip the description of the user update method and its test. They are basically the same as those for the create functionality. The DELETE /users/{id} endpoint provides a few different concepts, at least on the client side. We will flip the order with this one and show the client implementation first:

UserClient.groovy
boolean delete(final long userId) {
    http.delete {
        request.uri.path = "/users/$userId"
        response.success {
            true
        }
        response.failure {
            throw new IllegalArgumentException()
        }
    }
}

Notice the success and failure handlers used here. If you get a successful response (e.g. 200), the success handler is called, otherwise the failure handler is called. For our implementation, we want to return true if the delete is successful` and throw an IllegalArgumentException if the user was not deleted - yes, it’s a bit odd, but it shows a bit more functionality.

In order to test this method, we need to test cases:

UserClientSpec.groovy
def 'delete: successful'() {
    setup:
    server.expectations {
        delete('/users/42').called(1).responds().code(200)
    }

    UserClient client = new UserClient(server.httpUrl)

    when:
    boolean result = client.delete(42)

    then:
    result

    and:
    server.verify()
}

def 'delete: failed'() {
    setup:
    server.expectations {
        delete('/users/42').called(1).responds().code(500)
    }

    UserClient client = new UserClient(server.httpUrl)

    when:
    boolean result = client.delete(42)

    then:
    thrown(IllegalArgumentException)
    !result

    and:
    server.verify()
}

One test case tests the successful path and the other the failure case. While there is still a lot of functionality left to implement and test (e.g. more failure cases, bad input data, etc), we’ve got a good starting point and a framework for future testing.

Yes, this is a very code-rich discussion, but hopefully it was all pretty transparent about what was going on. You can find the code for both the client and the test in the rest-dev project on GitHub.

HttpBuilder-NG and Erstaz make a great team, and that’s actually somewhat by design. Ersatz is what HttpBuilder-NG uses to test its own functionality. Also, while the examples here are written in Groovy, both libraries work just as well with standard Java 8.

This post has only scratched the surface of the functionality provided by both libraries. Poke around their documentation and see what else you can do, and feature requests are always welcome.

Update: I have added a pure Java 8 implementation of the code for this post (source and tests). Yes, both libraries really do work well with Java too!


Creative Commons License CoffeaElectronica.com content is copyright © 2016 Christopher J. Stehno and available under a Creative Commons Attribution-ShareAlike 4.0 International License.