As seen previously, OVHcloud is a registrar. As a registrar, our work is very similar to that of a real estate agency. We connect tenants (registrants) with landlords (registries) and get a margin for each transaction. As a registrant, when you order a new domain name, you rent it for a given period (often 1 year), which can be extended until you don’t want it anymore.

In this article, we will show you what happens behind the scenes when you order and manage a domain name. Through a story, we will explain what the domain lifecycle is and how we interact with registries.

Lifecycle

Let’s say you read our blog article about the anatomy of GPUs using Pokémon, and want to start a business selling GPU Pokémon cards. You spent a lot a time thinking about the best domain name to register, and came up with pokegpu.com 

Our story starts on the 12th July 2021, you are ready to start your business. You go to the OVHcloud website and search this domain name.

In the following sections, for readability, periods of 30 and 60 days will be simplified respectively as 1 month and 2 months. It means that when 1st August is increased by 30 days, it will appear as 1st September and not 31th August. 

2021 – Domain Name Registration

First of all, when you search for pokegpu.com, OVHcloud information system will make a check to ensure that:

1. We sell the .com extension
2. There is no ongoing maintenance on the registry backend, Verisign in the case of .com.
3. The domain is available on the registry backend (also known as “Domain Check”)

Luckily, no one seems to want to sell GPU Pokémon cards, so you are free to order the domain name. Once the order is paid, we are able to start the creation of your domain name.

Depending on the extension, you may have to fulfill some requirements, like being a European Union citizen or resident for .eu extension (more on this in the next article). For the .com extension, no need to worry, the extension is available for everyone and there are no specific requirements.

On OVHcloud side, we will create the reference of your domain internally, and communicate with the registry backend by sharing some elements:

• Contacts data: owner, administrator, technical (more details about this in our next article)
• Dates of creation and expiration
• DNS servers (set by default or personalized in the order funnel)

In this creation phase, an anonymized working version of your email is sent to the registry backend, so you can avoid spam by regenerating another if needed. 

If transmitted information is correct, the creation of your domain is finalized in a short time.

Finally, by default, we transparently protect the domain name from unwanted transfers, by sending a command to the registry backend. So you will need to deliberately disable the protection if you want to transfer your domain name to another registrar.

As this is a new business and you are not sure that the company is viable, you decide to disable the OVHcloud auto-renew. It means the domain will have to be manually renewed before expiration, otherwise it will be deleted. Don’t worry, you will be notified before it happens.

It will require a manual renewal before 12th July 2022.

2022 – First Domain Renew

We are now 10 months later (60 days before expiration), the 12th May 2022. Nice, your business works well and you sell your PokeGPU cards all around the world .

But pokegpu.com is almost expired, so you receive a first notification. As you want to keep selling cards, you renew the domain name the 13th May 2022 for 2 years.

A renewal command is sent by our system to Verisign which adds two years to the expiration date.

It will now expire the 12th July 2024.

2024 – Forgotten Renewal

Two years have passed, you now have refocused on another business leaving the sale of GPU Pokémon cards aside for now. And in doing so, even after all the notifications (60 days, 30 days, 15 days, 7 days and 3 days prior to the expiration), you forgot to manually renew your domain name. The domain name expires the 12th July 2024.

This domain name, as well as the other gTLDs, are by default automatically renewed by the registry using the “registry auto-renew” feature. When the domain name expires, it’s renewed on the registry side, and an auto-renew grace period starts for up to 45 days, during which registrars can request the deletion of the domain name so that they are not charged for the renew.

It has to be differentiated from the OVHcloud auto-renew feature. OVHcloud auto-renew is possible on gTLDs extensions but also on the ones which do not support registry auto-renew. Here is a matrix to summarize which cases are possible and how the workflow changes.

Let’s come back to what happened for the expiration of your domain pokegpu.com:

1. As a registrar, when the expiration date was reached, we sent a command to suspend the domain name to the registry. This made the domain name unavailable, as well as the services hosted behind it (email, website…)
2. As it’s a gTLD, we waited for 30 days to let you renew the domain, in order to give you a the opportunity to renew during the auto-renew grace period
3. As it was not renewed during this period, we sent an explicit command to request deletion of the domain name to the registry. The domain name then fell in redemption grace period for 30 days. During this period, the domain name can still be restored from the registry

Now you have two choices:

1. You let it expire and 5 days after the end of redemption, it will be available again for registration by anyone (even by a malicious person)
2. You absolutely do not want to take the risk of losing the domain name and ask for a restore

2024 – Restoration

The domain is suspended and almost lost, it will return to the public domain in a couple of days. Grace period of 30 days expired, but the domain name can be restored. You don’t want to take the risk of having it taken over by someone else, so after some days of thought, you decide to keep your domain name for a little longer and explicitly restore the domain on the 20th August 2024.

This action is very similar to the renewal, but registries often ask for a much more expensive price (about 400% increase for .com). 

A restore command is sent to Verisign, which adds one year to the expiration date and reactivates the domain name. After the operation, the domain name is fully working and related hosted services become available again.

2024 – Secure renew with OVHcloud auto-renew

To avoid such trouble, at OVHcloud we decided to enable the OVHcloud auto-renew feature by default for new domain name registrations.

On the 21st August 2024, you decide to re-enable it. As long as a valid payment method is configured on your account, the domain name will be automatically renewed every year without any action on your end.

You can now focus on your new business with peace of mind  . The domain will be renewed on the 12th July 2025, the anniversary of the creation date, and in the following years until you decide not to. 

Summary

Here is a timeline summarizing what happened:

This timeline happened because a gTLD domain name lifecycle corresponds to the following simplified graph.

1. Domain name is available for registration to everyone
2. Domain name is registered for 1 to 10 years. The domain can be renewed by the customer in this period. If the expiration date is reached without renew, the domain name expires
3. Domain name is expired. It’s suspended and services hosted behind it are not available anymore. When the registry auto-renew feature is activated (to not mix up with OVHcloud auto-renew), registrars have up to 45 days to cancel the automatic renewal. At OVHcloud, we give customers 30 days to renew
4. Domain name is restorable. It’s still suspended and can be restored, which is similar to a renew but with redemption fees
5. Domain name is blocked in deletion for 5 days. It’s still suspended, and domain name will be available again for registration at the end of the period

Conclusion

In this post, we talked about the lifecycle of a single domain name. But the truth is that this story only relates to the lifecycle of a gTLD. ccTLDs belong to a whole new world, often with eligibility rules and a custom lifecycle. 

For instance, .es domain names don’t support automated restore, .it domain names have a renew grace period of 13 days, .pl domain names are not renewed on transfer… All these are special cases we need to integrate in our information system. As as registrar, our role is to lessen these differences by giving a standardized user experience to our customers.

In the next article, we will talk about eligibility rules and how some extensions cannot be owned by anyone.

At OVHcloud, the Domain Names team currently manages a total of 5 millions domain names. To handle this volume, and anticipate future growth, we have been migrating our information system from a monolith to a microservice-inspired architecture.

Although this ongoing migration presents lots of advantages over the monolithic approach (scalability being the most obvious), a few challenges arise as we progress towards our goal. By their nature, microservices imply a lot more network messages (HTTP in our case) and have a lot more dependencies on external services; such as databases, caches, queues, etc. In this context, unit testing each service individually is not enough. Testing the integration between all the services becomes critical.

We need to be able guarantee that the deployments have no impact on what had worked previously, and that new features work as expected. That’s why we focus heavily on integration and functional tests. To write efficient and relevant tests, it’s necessary to use appropriate tooling. In our case, we need tools matching our needs:

• Automated tool chain. This is an obvious one; but automating tests is a huge time saver as it allows us to run them automatically on our continuous integration platform at every single change. It also helps us building an ever-growing suite of tests which act as a non-regression chain.
• Easy-to-launch integration tests. Making the integration tests easy-to-launch, even locally, makes the team more likely to actually launch them, update them, and write them as a safety net. Even tests that require multiple interacting services (databases, queues, external APIs) must be easy to launch.
• Easy-to-write integration tests. Developer experience is important, the team is more likely to write tests if it’s easy.

Testing at the appropriate scale

In addition to regular unit tests, we run two kinds of API tests. This gives us the granularity and coverage we need for a safe deployment

At OVHcloud, the domain services are deployed as follows:

The services communicate through a central HTTP gateway, whose role is to manage the routing of calls between services using HTTP headers and ensure security. Every single HTTP call between the services goes through the gateway.

As these services expose APIs through HTTP endpoints, API testing is the perfect way to test them. For this to work effectively, the services should be treated like black boxes and the tests should only use their APIs. But, in reality, grey box testing is utilized. This is because we know the services implementation details, and we inject the datasets directly into the databases before testing.

As described in the diagram above, if we want to test the service Service 1, we require an environment with Gateway and potentially Service 2 or Service 3 to be available. This quickly becomes difficult to set up; especially if it is required for all the services to be tested.

So to test the API of a service, we need a way to isolate it from the others. This is what we will call isolated service tests in the rest of the blog post.

Isolated service tests

Isolated service tests are a kind of test between the unit test and the integration test. The service is started in a controlled environment, and its APIs are tested through various means. In isolated service tests, we allow for the following actions:

• perform API calls or send events,
• manipulate databases,
• make assertions using:
  • the responses to the API calls,
  • the events dispatched in queues,
  • direct queries on databases.

To do so, we launch the services with its technical dependencies (databases, queues, caches, etc) but we replace the HTTP gateway with an HTTP mock server. This approach helps us to avoid the deployment of other services and allows us to simply mock them.

Our tests are executed by a test runner. Before executing the actual tests, the role of the test runner is to populate the data of the environment:

• databases are populated using the “clean, insert” strategy,
• HTTP mocks are registered to the mock server.

This allows us to focus our tests correctly. In order to test a service, we need to set the state of the database and external services responses.

These tests are automatically executed at each merge on the staging branch of a service, and they serve as functional non-regression tests.

While this method is useful, it is not sufficient. As the calls are mocked, they must be updated manually at each change of the services involved. Also the mocks may not strictly correspond to reality. This is why we couple it with more complete integration tests.

Integration tests

The goal of these tests is to deploy all the services managed by the Domain Names team in the same state as in production. We still use a HTTP mock server instead of the HTTP Gateway, because there are still some external services we want to be able to mock. But the services belonging to our team are not mocked, we deploy them and register them to the mock server as backends.

This way, we can connect all our services together, and mock only the calls made to external services.

We use this to test complex workflows consisting of multiple calls made to different services. As a concrete example, it allows us to test the order funnel workflow by making the same calls that are made by the order funnel.

But unlike the isolated services tests, these are only executed once a day. This is because it requires deploying all our services in a short-lived environment, directly on a continuous integration platform – which may take a while.

Executing the tests

Technical stack

The method described above is generic, and can be applied using any tools you need. The abstract building blocks we need are:

• a test runner,
• an HTTP mock server,
• a continuous integration platform.

Venom, a declarative test runner

Venom is a declarative integration testing framework developed in-house at OVHcloud. Venom provides primitives to make HTTP calls, manipulate databases and message queues, and much more. It also provides a powerful context for writing assertions. This is completely optional though, simple shell scripts would work just fine, although they are less expressive and more complex to write.

With Venom, test suites are written in YAML. This makes the test suites easy to read and easy to store alongside the source code of the tested service. A test suite is just a sequence of test cases composed of multiple steps. The steps are actions executed sequentially on the service or its environment, on which we can perform assertions.

name: Testing "Users" service
version: "2"
testcases:
    - name: Initialize database fixtures
      steps:
        - type: dbfixtures
          database: postgres
          dsn: "{{.postgres_dsn}}"
          migrations: ../../testdata/schemas
          folder: ../../testdata/fixtures
     
    - name: Try to retrieve data about user 313
      steps:
        - type: http
          method: GET
          url: "{{.service_url}}/api/v1/users/313"
          assertions:
            - result.statuscode ShouldEqual 200
            - result.bodyjson.id ShouldEqual 313
            - result.bodyjson.first_name ShouldEqual John
            - result.bodyjson.last_name ShouldEqual Doe
     
    - name: Try to update the name of user 313
      steps:
        # Perform the update
        - type: http
          method: PATCH
          url: "{{.service_url}}/api/v1/users/313"
          body: |
            {
              "first_name": "Jane",
              "last_name": "Smith"
            }
          assertions:
            - result.statuscode ShouldEqual 200
 
        # Check that the first name and last name were correctly updated
        - type: http
          method: GET
          url: "{{.service_url}}/api/v1/users/313"
          assertions:
            - result.statuscode ShouldEqual 200
            - result.bodyjson.id ShouldEqual 313
            - result.bodyjson.first_name ShouldEqual Jane
            - result.bodyjson.last_name ShouldEqual Smith

The above test suite performs basic tests on a “Users” CRUD service. The test suite itself doesn’t start the service nor its dependencies, it assumes that they’re already started.

First, fixtures are loaded into the database. This is helpful to start testing most endpoints without having to manually register them through the API. Then, a few calls are performed and assertions are checked to ensure everything works as expected.

The killer feature of Venom is that it embeds a whole lot of executors that might be needed to test any service with arbitrary dependencies: raw scripts, HTTP, gRPC, SQL (MySQL and PostgreSQL), Redis, Kafka (producer and consumer), RabbitMQ, and even SMTP and IMAP to test out email sending!

Smocker, a HTTP mock server

As seen above, we need a server that can allow us to mock HTTP responses and simulate the behavior of a HTTP gateway.

Venom allows us to make assertions on the return values of HTTP calls and on the state of technical dependencies. But as it doesn’t provide the features we need alone, we fallback to an other tool, Smocker, developed for our use case. It features a user interface that is invaluable for writing mocks iteratively.

Thanks to Smocker, we can also perform a few more critical assertions on the internal behavior of our service:

• assert that the correct endpoints are called,
• assert that they’re called the right number of times,
• assert that they’re called with the right payload.

These features help us get a firmer grasp on the internals of the services we test, and help us to keep up with the evolution of the execution flow.

First, mocks must be registered to them via an API. A mock is simply a configuration file that initiates the sending of a given HTTP response when a specific call is made to an endpoint. The mock can have multiple filters; such as method, path, query parameters, etc. It also contains some contextual information; including the maximum number of calls on a route, response delay, etc.

- request:
    method: GET
    path: /users/313
    headers:
      X-Service-Destination: users-service
  response:
    status: 200
    headers:
      Content-Type: application/json
    body: |
      {
        "id": 313,
        "first_name": "John",
        "last_name": "Doe"
      }

Smocker can define mocks redirecting calls to another destination. This allows it to simulate the way the HTTP gateway works by defining the right mocks.

CDS, a continuous integration platform

CDS (Continuous Delivery Service) is a full-featured continuous delivery and automation platform developed in-house at OVHcloud. It’s a powerful alternative to other existing platforms; such as Jenkins, Travis or GitLab-CI. CDS allows us to create complex execution workflows running in various environments (virtual machines, containers).

CDS provides a concept of requirements which we use extensively to instantiate the expected environment for running our tests. This feature allows us to quickly declare the technical services that we will need during tests execution (databases, queues, caches, etc). It is quite similar to the “services” section available in a Travis file, for example.

Putting everything together

The tests are stored alongside the source code of services generally following this hierarchy:

tests/
  venom/
    schemas/
      .sql
      ...
    fixtures/
      .yml
      ...
    mocks/
      test_suite.mocks.yml
    test_suite.yml

For each service, the goal is to be able to launch the tests locally. The steps include:

• set up technical dependencies (databases, caches, etc) and the mock server within Docker containers,
• use variables in Venom tests files to be able to manipulate them with Venom (initialize data-sets, fill cache, etc),
• designate the mock server as our HTTP gateway using environment variables at service launch,
• fill Venom variables at test launch.

A Venom test suite could look like this:

version: "2"
name: Contacts endpoints testsuite
testcases:
- name: Save a new contact successfully
  steps:
  - type: dbfixtures
    database: postgres
    dsn: "{{.pgsql_dsn}}"
    migrations: ../schemas
    folder: ../fixtures
  - type: http
    method: POST
    url: "{{.mock_server}}/reset"
    assertions:
     - result.statuscode ShouldEqual 200
  - type: http
    method: POST
    url: "{{.mock_server}}/mocks"
    bodyFile: ./mocks/create_contact_mocks.yaml
    assertions:
     - result.statuscode ShouldEqual 200
  - type: http
    method: POST
    headers:
      Content-Type: application/json
    url: "{{.my_app}}/contacts"
    bodyFile: contact_create.json
    assertions:
     - result.statuscode ShouldEqual 201

Venom will:

1. initialize the database using the migration files and the fixtures available into /schemas and /fixtures,
2. reset the mock server,
3. set the mocks into Smocker (mock server) using the create_contact_mocks.ymlfile,
4. call the service using contact_create.json file as payload, and make assertions on the result.

To launch this test, all we need is to define at launch these Venom variables:

• pgsql_dsn: connection URL to the Postgres database,
• mock_server: administration URL of the mock server,
• my_app: URL of the service to test.

These variables are automatically set up within the Makefile of the service.

Launching functional tests locally allows us to debug our services step by step, and in a coherent context. As the datasets are controlled, it is quite easy to reproduce the errors and build efficient non-regression suites.

Once committed and merged on the staging branch, these tests are executed automatically by CDS. It works quite similarly to the local execution except that it is CDS and not a script that will set up the technical dependencies. The service is also launched as a container, using an image built in a previous job.

As indicated above, the technical dependencies, as well as the service, are declared as a service in the requirement part of the CDS job.

This allows to fill in CDS, the Venom variables mentioned above like this:

• pgsql_dsn: postgres://myuser:password@postgres/venom?sslmode=disable (user, password and database are set using options on requirement)
• mock_server: http://mockserver:8081 (default admin port on smocker)
• my_app: http://myapp:8080 (default service port)

Thanks to Venom’s command line, the only remaining thing to do in the pipeline is to execute the tests.

The testing pipeline is included in each of our workflows, and the deployment in production is necessarily blocked if any of the tests fail.

Retrospective

While setting up these integration tests, we faced some difficulties:

• The time to production was longer. Integration tests being executed before any deployment prevented us from deploying hotfixes instantaneously, they have to run through the integration tests beforehand. This is even worse if a hotfix breaks an integration test. Fortunately, in this case, it is possible to manually force the deployment through CDS.
• Integration tests are often quite complicated to maintain. But the YAML format of the tests and mocks allow us to comment to explain the content of these, which facilitates maintainability.

Aside from this, the overall experience with this testing method is extremely positive.

It adds an additional safety net to ensure the functional behavior of services as non-regression tests. Moreover, it allows us to be able to test new functionalities in advance, over several services, through updated mocks. And finally, it greatly eases reproducing functional bugs and debugging them.

The cost often associated with integration tests can be off-putting, but the implementation can easily be done incrementally on a service-by-service basis.

To go even further, we are now thinking of making several improvements.

Code coverage

Our micro-services are mainly written in Go. Go makes it possible to build a test binary – including a code coverage profile. This can be used in addition to Venom tests to generate the coverage of integration tests.

Code coverage is a nice-to-have metric as it enables you to detect sections of code that are less tested.

Documentation generation

Because Smocker, the HTTP mock server, is central to networking between the services, it has the ability to generate sequence diagrams describing the behavior of each call. This can be used to dynamically generate some sections of our internal documentation.