In previous articles, we saw what are the operational and techniques constraints of the migration project of our data center in Paris.

If you have not followed everything on our technical constraints, I invite you to reread the article presenting the infrastructure of our web hosting. We built our migration scenarios by taking these constraints seriously.

To overcome this high-risk project, we have considered several scenarios, each with their own set of operational difficulties and risks. Let’s take some time to see which migration projects we studied, and then we will explain how we selected the best one.

Scenarios

Our main concern in all migration scenarios was to avoid the split brain problem. This problem happens when the system receives simultaneously data writes in both the source of the migration and the destination.

Let’s take an example: an e-commerce site being migrated, available at the source and destination at the same time. If a customer of this site places an order, and this information arrives on the destination infrastructure, and that when he pays his order, the request arrives on the source infrastructure, the website can not make the link between the payment and order. This is what we call a split brain.

To solve this type of problem, it would be necessary to harmonize the two databases, which is only possible when one controls the model of data, and thus generally the source code of the site.

As a web hosting, we do not have the hands on the source code of our customers. And at our scale, we can not even imagine having to solve the problems we would encounter. Thus, we cannot consider any scenario involving split brain.

Migrating sites independently

Our first idea was to migrate web sites independently of each other. This is also the solution that we recommended for customers wishing to quickly benefit from the benefits of Gravelines before embarking on this global migration.

Here is what our customers usually had to do:

Identify all the databases used in their source code
Configure the destination account
Put the site in maintenance in the Paris data centre to avoid the split brain
Migrate all the data: source code files, as well as the databases
Configure the new database credentials in the site source code
Check the website works as intended
Modify their DNS zone in order to redirect their website to the new IP address of the new Gravelines Cluster
Re-open the site and wait for the end of the DNS propagation delay.

We have considered industrializing this technique to do it on behalf of the customer. But we were faced with several technical problems:

The reliable identification of all the databases used is complex. It is possible to search all database names in the source code of our customers, but it’s a very long operation, reliable only if the source code does not change during this time.
It also requires dealing with many special cases: binary files executed in CGI, source code obfuscation, or even storage of the name of the databases in… a database. This technique does not allow us to consider a 100% reliability of our migration.
The migration of the files can be done according to two techniques:
1. In file mode,, the script of migration traverses the file tree and copies them one by one.
2. In block mode, the migration script takes the data from the hard disk and transfers it bit-by-bit to the destination hard disk without taking into account the tree structure.

Both methods allow you to copy the data reliably, but the intended use case is very different,

With block mode you only can copy an entire disk or partition. If on the same partition there is data from several websites, only file mode allows to migrate a website data individually.

Moving the data in file mode is very slow if the number of files to browse is important, as it’s the case for many PHP frameworks performing caching. So we had the risk of being unable to migrate some sites.

Modifying the source code is a perilous operation that we do not allow ourselves to do because the impact on the website can be significant. In addition, it requires to have exhaustively identified all uses of databases …
A number of our customers do not host their DNS zone at home. We are then unable to change the IP address of their website without their intervention, which requires us to keep this IP address if we want to achieve a good level of reliability for the migration.

We have therefore declined this scenario. Although functional for a large majority of websites, the small percentage of sites that would have been impacted represents actually a large number of websites. We should have repaired them manually and our team would have spent all his time there.

IP over Truck Carriers

Internet is based on the IP protocol for addressing machines across the network. It is not dependent on the physical material on which the message is exchanged, it is possible to use many: optical links, electrical, wireless; even traveling pigeons, as described by a humorous standard set on April 1, 1990!

This April Fool joke inspired us, although we are not experts in pigeons. Indeed, even if the latency (travel duration for a message from point A to point B) is important, the bandwidth (amount of information sent / travel time) is potentially huge: a USB key contains a lot of data ! In some large transfers, moving physically the data is a reasonable way to increase the bandwidth of a transfer.

So we thought about the option of simply moving the infrastructure from Paris to Gravelines. This has advantages:

No impact on websites. We just have to turn the infrastructure back on in another data center and redirect traffic to it;
It allows you to empty the data center very quickly.

But it also poses some challenges:

How to reduce the time of cutting websites between the shutdown of the machines in Paris, their loading, transport, unloading and re-ignition? The cut-off time would be of the order of several days.
What to do in the event of an accident on the journey? Falling server, road accident …
How do you make sure that the data will not be altered during transport because of truck vibrations?
How to integrate this infrastructure not respecting the standards of industrialization in force in Gravelines?

None of this points was blocking points but they raised interesting challenges. We therefore kept this scenario, although not as our first choice because of the physical risks and the long time of unavailability of the websites during the operation.

Scalable Migrations

Not being able to migrate the entire datacentre all at once, nor the websites independently, we have been looking at how to migrate our infrastructure assets as we go.

So we took a step back and looked at the levels of granularity of our infrastructure, that is, the elements that link websites to each other and prevent a site-by-site migration:

IP addresses: we did not control all the DNS zones, we considered that the IP addresses of the websites of our customers could not be modified. This means that we must migrate all websites using the same IP address at once.
The filerz: migration in file mode data on filerz not being possible because of the large number of files, we must perform a migration in block mode and thus migrate all customers in the same filerz simultaneously.
Databases: all the databases on the same website must be migrated at the same time to keep the site running, and database identifiers must not change. These databases can potentially be used by two different locations, including different clusters; these accommodations must be migrated at the same time.

If we consider all these assumptions, one conclusion is necessary: to respect them, we must migrate all sites at once because of interdependencies.

We were in a blocking position. To move forward, it was necessary to review each postulate, and consider solutions to overcome these problems.

Breaking dependencies on databases

One of the most complex constraints is to reliably migrate databases together with websites.

Could we imagine a 95% reliable migration, taking into account only the databases provided with hosting (that is, leaving aside atypical cases that are found only by analyzing the source code of websites)?

On paper it wouldn’t work, as the atypical websites would be impacted, since the databases would no longer be available.

We thus needed to play with the availability of the database: if we manage to keep an accessible database even if it is not migrated, we can remove this constraint, and atypical cases would continue to work.

This is technically possible if we open a network tunnel between our data centre in Gravelines and that in Paris. With this tunnel, a website using a database that is not referenced in its hosting would continue to work by fetching the data in Paris.

This is not the ideal solution: adding a network tunnel means adding 10ms latency. And on some CMS performing dozens of SQL queries sequentially, this latency is quickly visible. But by limiting this effect to only non-referenced databases, we could simplify this strong constraint. All the sites would continue to operate. Some sites might experience some slowness, but for our usual cases of web hosting, the repercussions are minimal.

Bypassing IP Address Dependency

Behind a single IP address are several hundred thousand websites. Migrating all filerz and databases would therefore involve a very important shutdown time.

However, the question can be asked differently: an IP address serves several sites, how can we distribute incoming traffic on that IP to the right data centre hosting the right website? It is a load balancing concern and we already have a load balancer that adapts according to the requested website: the predictor.

It is possible to define within the predictor where the website really is to redirect the traffic. The simplest solution would be adding a new predictor upstream of our infrastructure, but chaining load balances is not a good idea: it makes the path to the website more complex and adds a new critical element in the infrastructure.

And finally, nothing prevents us to use the load balancing of Paris or Gravelines to perform this traffic redirection.

We selected the predictor of our new Gravelines clusters. We added the list of websites and their status: migrated or not migrated. Thus, for migrated sites, the traffic is distributed locally. Otherwise, the traffic is redirected to a load balancer of the cluster in Paris.

We knew how to migrate an IP address between our data centres. It is therefore possible to prepare these new predictors and then migrate the IP addresses of a whole cluster in a transparent way, without causing a customer break.

IP addresses were no longer a blocking point. By lowering this constraint, we can now migrate customers filerz by filerz. Can we do even better?

Migrate the filerz as a block

To do better we would need to decorrelate the clients of each filerz. How could we do it?

Migrating an entire filerz takes time. We need to move several TBs of data across our network, it can take dozens of hours. To avoid split brain, we need to avoid writing at the source during the copy.

But our Storage team knows how to handle this type of case, rather common for them. They begin by making a first copy of all the data, without closing the source service. Once this first copy is made, latter copies only need to synchronize the differences written since the first one. After several successive copies, the copy time is very short.

At that moment, it is possible to cut the service for a few hours, at night, to perform the migration (filerz and the associated databases) without risking split brain.

We now have all the elements to realize a new migration scenario!

Come on, let’s do it again.

Our final scenario

After reading the precedent sections, you probably already have a good idea of how we migrate clusters. Here are the main steps:

1 / Construction of a new cluster in Gravelines, up to the standards of this new datacenter but including as many filerz and databases as the previous one.

2 / Building a network link between the new and the old datacenter.

3 / Migration of IP traffic on the new cluster with redirection to Paris for non-migrated sites.

4 / Copying data without breaking websites.

5 / At night: cutting off the sites of the first filerz, migrating its data and associated databases.

6 / At night: shutdown of sites, migration of second filerz and associated databases.

7 / Closing the source cluster.

Even after we have migrated all the filerz of a cluster, the network link to Paris data centre need to be kept up until all the databases have been migrated, at the very end of the migration. This link must thus be long-lasting and monitored for several months.

This scenario was validated in July 2018. To make it operational, it took us 5 months of adaptation of our systems, as well as multiple blank repetitions on a test cluster specifically deployed to verify everything worked as intended. The scenario looked pretty on paper, but we had to solve many problems at every step of the process (we will go deeper into the technical details in future blog posts).

Every step in this scenario implied dozens of synchronized operations between the different teams. We had to put in place a very precise follow-up of our operations to ensure that everything goes without a hitch (we will also talk about it in another post).

Now you know our migration process. This article, though long, is necessary to understand the architectural choices, and the surprises that we encountered during our migration.

In future posts, we will focus on specific points of our migration and the technical challenges we have encountered.

See you soon for the rest of our adventures!

Vincent Cassé

+ posts

Engineering manager on webhosting