At OVHcloud, the Domain Names team currently manages a total of 5 millions domain names. To handle this volume, and anticipate future growth, we have been migrating our information system from a monolith to a microservice-inspired architecture.
Although this ongoing migration presents lots of advantages over the monolithic approach (scalability being the most obvious), a few challenges arise as we progress towards our goal. By their nature, microservices imply a lot more network messages (HTTP in our case) and have a lot more dependencies on external services; such as databases, caches, queues, etc. In this context, unit testing each service individually is not enough. Testing the integration between all the services becomes critical.
We need to be able guarantee that the deployments have no impact on what had worked previously, and that new features work as expected. That’s why we focus heavily on integration and functional tests. To write efficient and relevant tests, it’s necessary to use appropriate tooling. In our case, we need tools matching our needs:
- Automated tool chain. This is an obvious one; but automating tests is a huge time saver as it allows us to run them automatically on our continuous integration platform at every single change. It also helps us building an ever-growing suite of tests which act as a non-regression chain.
- Easy-to-launch integration tests. Making the integration tests easy-to-launch, even locally, makes the team more likely to actually launch them, update them, and write them as a safety net. Even tests that require multiple interacting services (databases, queues, external APIs) must be easy to launch.
- Easy-to-write integration tests. Developer experience is important, the team is more likely to write tests if it’s easy.
Testing at the appropriate scale
In addition to regular unit tests, we run two kinds of API tests. This gives us the granularity and coverage we need for a safe deployment
At OVHcloud, the domain services are deployed as follows:
The services communicate through a central HTTP gateway, whose role is to manage the routing of calls between services using HTTP headers and ensure security. Every single HTTP call between the services goes through the gateway.
As these services expose APIs through HTTP endpoints, API testing is the perfect way to test them. For this to work effectively, the services should be treated like black boxes and the tests should only use their APIs. But, in reality, grey box testing is utilized. This is because we know the services implementation details, and we inject the datasets directly into the databases before testing.
As described in the diagram above, if we want to test the service Service 1, we require an environment with Gateway and potentially Service 2 or Service 3 to be available. This quickly becomes difficult to set up; especially if it is required for all the services to be tested.
So to test the API of a service, we need a way to isolate it from the others. This is what we will call isolated service tests in the rest of the blog post.
Isolated service tests
Isolated service tests are a kind of test between the unit test and the integration test. The service is started in a controlled environment, and its APIs are tested through various means. In isolated service tests, we allow for the following actions:
- perform API calls or send events,
- manipulate databases,
- make assertions using:
- the responses to the API calls,
- the events dispatched in queues,
- direct queries on databases.
To do so, we launch the services with its technical dependencies (databases, queues, caches, etc) but we replace the HTTP gateway with an HTTP mock server. This approach helps us to avoid the deployment of other services and allows us to simply mock them.
Our tests are executed by a test runner. Before executing the actual tests, the role of the test runner is to populate the data of the environment:
- databases are populated using the “clean, insert” strategy,
- HTTP mocks are registered to the mock server.
This allows us to focus our tests correctly. In order to test a service, we need to set the state of the database and external services responses.
These tests are automatically executed at each merge on the staging branch of a service, and they serve as functional non-regression tests.
While this method is useful, it is not sufficient. As the calls are mocked, they must be updated manually at each change of the services involved. Also the mocks may not strictly correspond to reality. This is why we couple it with more complete integration tests.
Integration tests
The goal of these tests is to deploy all the services managed by the Domain Names team in the same state as in production. We still use a HTTP mock server instead of the HTTP Gateway, because there are still some external services we want to be able to mock. But the services belonging to our team are not mocked, we deploy them and register them to the mock server as backends.
This way, we can connect all our services together, and mock only the calls made to external services.
We use this to test complex workflows consisting of multiple calls made to different services. As a concrete example, it allows us to test the order funnel workflow by making the same calls that are made by the order funnel.
But unlike the isolated services tests, these are only executed once a day. This is because it requires deploying all our services in a short-lived environment, directly on a continuous integration platform – which may take a while.
Executing the tests
Technical stack
The method described above is generic, and can be applied using any tools you need. The abstract building blocks we need are:
- a test runner,
- an HTTP mock server,
- a continuous integration platform.
Venom, a declarative test runner
Venom is a declarative integration testing framework developed in-house at OVHcloud. Venom provides primitives to make HTTP calls, manipulate databases and message queues, and much more. It also provides a powerful context for writing assertions. This is completely optional though, simple shell scripts would work just fine, although they are less expressive and more complex to write.
With Venom, test suites are written in YAML. This makes the test suites easy to read and easy to store alongside the source code of the tested service. A test suite is just a sequence of test cases composed of multiple steps. The steps are actions executed sequentially on the service or its environment, on which we can perform assertions.
name: Testing "Users" service
version: "2"
testcases:
- name: Initialize database fixtures
steps:
- type: dbfixtures
database: postgres
dsn: "{{.postgres_dsn}}"
migrations: ../../testdata/schemas
folder: ../../testdata/fixtures
- name: Try to retrieve data about user 313
steps:
- type: http
method: GET
url: "{{.service_url}}/api/v1/users/313"
assertions:
- result.statuscode ShouldEqual 200
- result.bodyjson.id ShouldEqual 313
- result.bodyjson.first_name ShouldEqual John
- result.bodyjson.last_name ShouldEqual Doe
- name: Try to update the name of user 313
steps:
# Perform the update
- type: http
method: PATCH
url: "{{.service_url}}/api/v1/users/313"
body: |
{
"first_name": "Jane",
"last_name": "Smith"
}
assertions:
- result.statuscode ShouldEqual 200
# Check that the first name and last name were correctly updated
- type: http
method: GET
url: "{{.service_url}}/api/v1/users/313"
assertions:
- result.statuscode ShouldEqual 200
- result.bodyjson.id ShouldEqual 313
- result.bodyjson.first_name ShouldEqual Jane
- result.bodyjson.last_name ShouldEqual SmithThe above test suite performs basic tests on a “Users” CRUD service. The test suite itself doesn’t start the service nor its dependencies, it assumes that they’re already started.
First, fixtures are loaded into the database. This is helpful to start testing most endpoints without having to manually register them through the API. Then, a few calls are performed and assertions are checked to ensure everything works as expected.
The killer feature of Venom is that it embeds a whole lot of executors that might be needed to test any service with arbitrary dependencies: raw scripts, HTTP, gRPC, SQL (MySQL and PostgreSQL), Redis, Kafka (producer and consumer), RabbitMQ, and even SMTP and IMAP to test out email sending!
Smocker, a HTTP mock server
As seen above, we need a server that can allow us to mock HTTP responses and simulate the behavior of a HTTP gateway.
Venom allows us to make assertions on the return values of HTTP calls and on the state of technical dependencies. But as it doesn’t provide the features we need alone, we fallback to an other tool, Smocker, developed for our use case. It features a user interface that is invaluable for writing mocks iteratively.
Thanks to Smocker, we can also perform a few more critical assertions on the internal behavior of our service:
- assert that the correct endpoints are called,
- assert that they’re called the right number of times,
- assert that they’re called with the right payload.
These features help us get a firmer grasp on the internals of the services we test, and help us to keep up with the evolution of the execution flow.
First, mocks must be registered to them via an API. A mock is simply a configuration file that initiates the sending of a given HTTP response when a specific call is made to an endpoint. The mock can have multiple filters; such as method, path, query parameters, etc. It also contains some contextual information; including the maximum number of calls on a route, response delay, etc.
- request:
method: GET
path: /users/313
headers:
X-Service-Destination: users-service
response:
status: 200
headers:
Content-Type: application/json
body: |
{
"id": 313,
"first_name": "John",
"last_name": "Doe"
}Smocker can define mocks redirecting calls to another destination. This allows it to simulate the way the HTTP gateway works by defining the right mocks.
CDS, a continuous integration platform
CDS (Continuous Delivery Service) is a full-featured continuous delivery and automation platform developed in-house at OVHcloud. It’s a powerful alternative to other existing platforms; such as Jenkins, Travis or GitLab-CI. CDS allows us to create complex execution workflows running in various environments (virtual machines, containers).
CDS provides a concept of requirements which we use extensively to instantiate the expected environment for running our tests. This feature allows us to quickly declare the technical services that we will need during tests execution (databases, queues, caches, etc). It is quite similar to the “services” section available in a Travis file, for example.
Putting everything together
The tests are stored alongside the source code of services generally following this hierarchy:
tests/
venom/
schemas/
.sql
...
fixtures/
.yml
...
mocks/
test_suite.mocks.yml
test_suite.ymlFor each service, the goal is to be able to launch the tests locally. The steps include:
- set up technical dependencies (databases, caches, etc) and the mock server within Docker containers,
- use variables in Venom tests files to be able to manipulate them with Venom (initialize data-sets, fill cache, etc),
- designate the mock server as our HTTP gateway using environment variables at service launch,
- fill Venom variables at test launch.
A Venom test suite could look like this:
version: "2"
name: Contacts endpoints testsuite
testcases:
- name: Save a new contact successfully
steps:
- type: dbfixtures
database: postgres
dsn: "{{.pgsql_dsn}}"
migrations: ../schemas
folder: ../fixtures
- type: http
method: POST
url: "{{.mock_server}}/reset"
assertions:
- result.statuscode ShouldEqual 200
- type: http
method: POST
url: "{{.mock_server}}/mocks"
bodyFile: ./mocks/create_contact_mocks.yaml
assertions:
- result.statuscode ShouldEqual 200
- type: http
method: POST
headers:
Content-Type: application/json
url: "{{.my_app}}/contacts"
bodyFile: contact_create.json
assertions:
- result.statuscode ShouldEqual 201Venom will:
- initialize the database using the migration files and the fixtures available into /schemas and /fixtures,
- reset the mock server,
- set the mocks into Smocker (mock server) using the create_contact_mocks
.ymlfile, - call the service using contact_create.json file as payload, and make assertions on the result.
To launch this test, all we need is to define at launch these Venom variables:
- pgsql_dsn: connection URL to the Postgres database,
- mock_server: administration URL of the mock server,
- my_app: URL of the service to test.
These variables are automatically set up within the Makefile of the service.
Launching functional tests locally allows us to debug our services step by step, and in a coherent context. As the datasets are controlled, it is quite easy to reproduce the errors and build efficient non-regression suites.
Once committed and merged on the staging branch, these tests are executed automatically by CDS. It works quite similarly to the local execution except that it is CDS and not a script that will set up the technical dependencies. The service is also launched as a container, using an image built in a previous job.
As indicated above, the technical dependencies, as well as the service, are declared as a service in the requirement part of the CDS job.
This allows to fill in CDS, the Venom variables mentioned above like this:
- pgsql_dsn: postgres://myuser:password@postgres/venom?sslmode=disable (user, password and database are set using options on requirement)
- mock_server: http://mockserver:8081 (default admin port on smocker)
- my_app: http://myapp:8080 (default service port)
Thanks to Venom’s command line, the only remaining thing to do in the pipeline is to execute the tests.
The testing pipeline is included in each of our workflows, and the deployment in production is necessarily blocked if any of the tests fail.
Retrospective
While setting up these integration tests, we faced some difficulties:
- The time to production was longer. Integration tests being executed before any deployment prevented us from deploying hotfixes instantaneously, they have to run through the integration tests beforehand. This is even worse if a hotfix breaks an integration test. Fortunately, in this case, it is possible to manually force the deployment through CDS.
- Integration tests are often quite complicated to maintain. But the YAML format of the tests and mocks allow us to comment to explain the content of these, which facilitates maintainability.
Aside from this, the overall experience with this testing method is extremely positive.
It adds an additional safety net to ensure the functional behavior of services as non-regression tests. Moreover, it allows us to be able to test new functionalities in advance, over several services, through updated mocks. And finally, it greatly eases reproducing functional bugs and debugging them.
The cost often associated with integration tests can be off-putting, but the implementation can easily be done incrementally on a service-by-service basis.
To go even further, we are now thinking of making several improvements.
Code coverage
Our micro-services are mainly written in Go. Go makes it possible to build a test binary – including a code coverage profile. This can be used in addition to Venom tests to generate the coverage of integration tests.
Code coverage is a nice-to-have metric as it enables you to detect sections of code that are less tested.
Documentation generation
Because Smocker, the HTTP mock server, is central to networking between the services, it has the ability to generate sequence diagrams describing the behavior of each call. This can be used to dynamically generate some sections of our internal documentation.
Developer @ Domain Names Squad.
Developer @ Domain Names Squad.