For startups ready to scale smarter, OVHcloud offers a compelling alternative: high-performance, cost-effective, and sovereignty-first infrastructure. Here’s why more and more growing startups are making the switch.

1. Predictable, Transparent Pricing

OVHcloud’s flat-rate pricing model eliminates hidden fees and unpredictable billing. Bandwidth is included. Egress costs? Zero. This gives startups the ability to budget confidently—even as infrastructure scales rapidly.

2. Cost-Efficient Scaling

Startups that migrate to OVHcloud often report up to 60% cost savings compared to hyperscalers. Whether you’re scaling your backend, AI workloads, or customer-facing applications, OVHcloud lets you do more with less.

3. Performance Without Compromise

From Bare Metal servers and high-memory VMs to GPU and storage-optimized instances, OVHcloud infrastructure is engineered for performance. Ideal for AI, SaaS, analytics, and other compute-intensive use cases.

4. Full Data Sovereignty in the EU

OVHcloud is headquartered in Europe and operates under the strictest data protection laws (like GDPR in the EU or Law 25 in Quebec, Canada). Unlike other providers, your data stays within jurisdictions that respect privacy and sovereignty—no exposure to foreign surveillance laws.

5. Open Standards and No Vendor Lock-In

Freedom matters—especially when you’re building for scale. OVHcloud supports open technologies like Kubernetes, Terraform, and OpenStack, giving your team full flexibility and avoiding lock-in to proprietary tools or services.

6. Infrastructure That Grows With You

Whether you’re launching in new markets or onboarding thousands of new users, OVHcloud enables seamless horizontal and vertical scaling. With availability across multiple regions, your growth won’t hit a wall.

7. Faster Time-to-Market Through Cloud Migration Support

OVHcloud offers cloud migration guidance and tools, including compatibility with major platforms and migration kits. This helps your team move faster, avoid downtime, and focus on innovation—not infrastructure headaches.

8. Dev-Friendly Ecosystem

With support for containerization, automation, and CI/CD pipelines, OVHcloud makes life easier for DevOps teams. You can provision infrastructure programmatically and scale efficiently—just like you would with AWS or GCP.

9. Sustainability Built In

Efficiency is built into OVHcloud’s DNA. By designing and operating its own energy-efficient data centers, OVHcloud helps startups meet their sustainability goals without compromising on performance or cost.

10. A Cloud Partner—Not Just a Provider

Startups need more than compute power—they need guidance, flexibility, and a partner that understands their journey. OVHcloud offers tailored support, technical documentation, and real human engagement to help you succeed at every stage of growth.

OVHcloud: Built to Scale With Startups

If your startup is growing fast and needs infrastructure that can keep up—without breaking the budget or sacrificing control—OVHcloud offers a cloud built around your values: scalability, transparency, freedom, and performance.

Migrate with confidence. Scale with control. Grow with OVHcloud.

If you’re a startup looking to transform your business, we encourage you to join the OVHcloud Startup Program or contact OVHcloud to discover how our solutions can support your journey!

Based on the current industry business model there are two types of predominant words when we speak with CIO: Multi-Cloud and/or Hybrid Cloud approach.

What we really intend with Multi-Cloud approach?

• A multi-cloud approach is a strategy that involves the company using multiple CSPs to meet the IT and HW needs of the company.
• This approach is a valid approach for companies that would like to differentiate CSPs services using some specific services by different Cloud Providers; considering the variety of services available in the market, the approach, to move from internal IT/HW to external Cloud Providers can be very effective with an increased flexibility to meet the needs of the company and customization of some services;
• Another advantage is about Cost Optimization as the Multi-Cloud approach can help the organizations to differentiate the services by optimizing the costs based on the choice of the most cost-effective services provided by the Cloud Providers.
• furthermore, the choice of Multi-Cloud could help to reduce risk of outages by differentiating the services over multiple service providers.

Based on the above statements it looks clearly that Multi-Cloud approach represents the future evolution of the traditional Cloud Computing.

The challenges of Multi-Cloud

There are challenges that the company should evaluate deeply before moving in a multi-cloud approach such as:

• Ensure the Interoperability between the CSPS, this is a key element to have a clear views of who is doing what, to avoid bad surprises the company should have a control about the IT services running over different CSPs and this requires to have skilled person able to operate above the HW layer and understand how orchestrate at best the various IT services.
• Security and compliance is another delicate aspect of this approach as the company should state which data can operate in compliance with the regulation, as per example Health Data must be secured under GDPR and hosted in DC with specific certifications to secure their data.

It is evident that in a Multi-Cloud approach the (System Integrators) SI and Managed Services Providers (MSP) play a key role in supporting and guiding the organizations to choose and manage multiple cloud environments.

Nowadays the shortage of the skills are forcing companies to delay the cloud adoption, however choosing the right SI/MSP it’s a must move for: designing, deploying and in certain case operating the platform on behalf of the customers.

The Ecosystem of Partners

The Ecosystem of OVHcloud have a huge variety of SI and MSPs that can help and support the organizations to move faster in direction of the new Cloud approach with the right skills to avoid delay in Cloud adoption and move the IT faster in a new era of fast Business needs.

We again hugely invite the companies that are looking for the right cloud approach to go in the Partner Directory of OVHcloud and find the right partner for their exigences.

The benefits for clients to shift their applications and IT landscapes to the cloud are already well understood. It usually revolves around expected performance gains, scalability advantages of cloud-native products, boosted productivity, security and greatly reduced maintenance efforts for all assets leveraging cloud platform services. In summary the ultimate goal is shifting from a CAPEX model to OPEX for more IT infrastructure and technology agility and lowered system level maintenance.

But on the other hand cloud transformation projects require large investments, have extensive runtimes, and heavily bind both client internal teams and resources as well as advisors and migration teams coming from external consulting.

Because of the complexity, size and impact of a typical cloud transformation project, decision makers either in industry and or public sector are very careful about deciding for a cost effective cloud platform that meet their needs.

In order to build up that confidence, decision makers need to have a clear vision and insight on the market and approach migrating its estate over to their cloud platform of choice.

To succeed, cloud transformation projects need a clear vision

When an organization initiates a migration of their on-premise infrastructure to the cloud, the need to modernize their applications (or at least a subset of the applications) is generally clear. However, most organizations have a hard time picturing the future state of their IT infrastructure, once migrated to the cloud. 

And yet, migrating an application is much smoother when you already have a precise idea of what the final infrastructure will look like (for instance: which cloud services will be included, what will be the overall costs, etc.). 

Having this depth of knowledge is essential from a project management perspective. In order to accelerate the migration process, and to minimize risk, organizations need to share a clear vision of the end result. 

Moving to the cloud is also a time of uncertainty, especially for organizations that used to run all their applications and workloads on-premises. Thus, the success of your project will largely depend on the ability of managers to pave the way. “This is where we are at the moment”. “This is our destination”. “And here are different ways to get there”. Providing visibility on what the future infrastructure will look like is key to reduce fear and empower the teams to achieve the transformation process. 

A clear vision also comes with precise goals. What is the business case behind this application? There are many reasons to move to the cloud. Some organizations want to modernize their infrastructure from the ground up and are ready to spend several years on it. Others aim to stop running their hardware environment in the short term. Finally, for others, cost savings will be the decisive criterion. The optimal cloud architecture will largely depend on the organization’s goals and priorities. 

The hard time of comparing cloud offerings

Ideally, when planning the migration of an application, an organization should proceed as follows: 

• Define the goals of the migration
• Compare different migration alternatives
• Model the exact cloud architecture that is needed for this application

In practice, however, this can quickly become a labor intensive task, and one reason for that is the difficulty to compare the options available on the market. It’s no understatement that there is a wide range of cloud offerings, and that companies are spoilt for choice. But the main difficulty comes from the fact that each supplier has its own way of presenting its services and its pricing model. 

In the end, organizations miss a clear comparison table to efficiently investigate the pros and cons of each option, and make their migration decisions with speed and confidence. 

Cloud Insider: a knowledge base for multi-cloud market insights

To empower organizations on their transformation journey, Txture developed Cloud Insider, a solution that provides an unparalleled overview of the fast changing cloud market, and simplifies cloud services comparison and price forecasting. 

Cloud Insider knowledge base includes products from the vast majority of cloud vendors, and maintains up-to-date information about hundreds of products, and hundreds of thousands of available product variants. 

The goal of Cloud Insider is to provide cloud professionals with a comprehensive and provider-neutral overview of the cloud market. Leveraging Cloud Insider, you can quickly compare similar products based on several criteria (features, certifications, location, pricing model, etc.) and make your migration decisions much faster. 

How Cloud Insider works

In the following section, we’ll see how Cloud Insider can be used to quickly model cloud architecture alternatives. In this example, we’ll assume that your organization wants to migrate one of its workloads to OVHCloud. Your workload consists of a server, a database and a storage solution. 

1 – Modelling a cloud architecture

Browsing the knowledge base, you will be able to quickly find a server solution that fits your requirements. To narrow down your research, you can filter for specific results. For instance, you want the server to be located in the UK, and you also want the solution to be GDPR compliant.

You are presented with two different products of the OVHcloud catalog that fit the requirements. When you click on one product, you will see a list of product variants. You can again narrow down the list with additional filters, such as number of CPUs, operating system, instance type, storage type, etc.

You can proceed similarly to find a managed database and a storage solution. Once you have selected all the components you need, Cloud Insider generates an overview of the cloud architecture you just created. The platform also provides you with a bill of materials, so you get a clear idea of the overall costs of the architecture. 

For more and more organizations, sustainability has become a key criteria when choosing a cloud infrastructure. In the overview provided by Cloud Insider, you can also find insights into the carbon footprint of your current scenario. The platform automatically calculates the monthly energy consumption and CO2 emissions of your target architecture, as you can see below: 

2 – Comparing migration scenarios

Before making your migration decision, you probably want to consider a few alternatives. This is where the Cloud Insider platform can accelerate the comparison of different cloud architecture options. Once you created a first cloud architecture, you can build alternatives and test different product variants, locations, storage types, etc. 

Then, the platform will offer you a side-by side comparison. You will be able to quickly compare different cloud architectures based on pricing model, features, but also certifications and carbon footprint. In the end, you save time, and you can make your migration decisions based on up-to-date information. 

In this example, we compare different cloud architectures on OVHcloud, but you can also use it to compare architectures of different cloud vendors. 

The Bottom Line

We all know oftentimes it is hard to keep track of cloud services offered today and those added every month. How to compare services across providers, in which data center locations are services available, and what price tags do they have attached? Still as a cloud professional it is key to be confident with your decisions in order to pick the right solution from a pool of different options out there.

With Cloud Insider, browse cloud services by provider, location, certifications, and underlying technology. Rely on detailed analysis and up-to-date data of cloud products from various vendors to select the right services based on your requirements.

View cloud service product instances, certification details and compare to similar products of other providers. Easily filter instances by location, CPU, RAM, OS and compare on-demand vs. reserved instance prices.

With Cloud Insider cloud professionals finally can take out the uncertainty exploring the cloud market and can make educated decisions.

To learn more go to: https://cloudinsider.com

Migration to the cloud offers countless advantages and seems to be an inevitable step in the development of any modern enterprise. Those advantages become clearly visible in most sectors, especially finance and insurance, where applied solutions must both protect the customer data and offer cost-effective ways of handling big chunks of information.

Many customers are anxious to know where their data is going. How can they be sure the data is safe? What happens if the data disappears? Or gets stolen?

Sensitive data needs secure infrastructure, but time-effective solutions are also key to ensure success in any competition-dense industry.

Cloud migration requires some specific skills and experience that aren’t always available in-house. That’s what OVHcloud Partners can help you with.

Let’s have a look how two OVHcloud partners, Crowtech and Zapaśnik IT joined forces to create an innovative and secure solution for an insurance company by leveraging OVHCloud BareMetal Cloud. They created a distributed virtualization solution across PL/DE data centers with their own managed Kubernetes.

By implementing this solution, the customer gewas able to become a leader of innovation on the market and take over larger contracts, that until now where unreachable.

A customer looking to migrate to the Cloud

The Customer, Promesa Plus, is a broker offering the vehicles insurance policies from all top players on the market. The company’s target market are car dealers, as well as corporate customers.

Since having introduced their first solution in 2004, Promesa Plus has been the most innovative company on the market, moving ahead of its competitors. However after several years, the system they built needed an upgrade.

The broker was looking for a solution to address numerous challenges, such as fast-evolving insurance rates, complexity and variety of products, as well as numerous algorithms to calculate the final rates.

Managing an insurance enterprise is not an easy job, given that data management and security are essential. The initial on-premise infrastructure was not sufficient and they were not able to implement the industry regulations, security best practices and system development needs.

Because of these limitations, the business was underperforming – updating the database, adding a new customer/partner or using the e-commerce system were taking too much time, leading customers to leave their baskets abandoned.

Unfortunately, any modernization of the system was also restricted, as the development cycle was not automated. For example, it could only populate new extensions to system in a “traditional” way. Any new code required a reboot of the system. This was extremely time-consuming and ended up leaving room in the marketplace for the competition.

Consequently, Promesa Plus could not easily process both growing number of transactions and integration of new customers. As a result, both employees and customers had a negative experience and this in turn has put the growth of the company in jeopardy.

A strategic decision was made to direct the upcoming transformation towards efficiency, scalability, security and availability of the system. To support the company in this challenging task, Promesa Plus invited two partners Crowtech & Zapaśnik IT to join them on this journey. Having researched many potential solutions thoroughly, Promesa Plus chose to pursue the Cloud’s advantages.

They decided to migrate the whole environment, including the development, quality and production with CI/CD pipeline, built new e-commerce and put in place a disaster-recovery plan.

The solution

The solution they came up with was a microservices-oriented architecture built on the OVHcloud BareMetal Cloud using Kubernetes orchestration turned out to be the best solution for the challenges and specificity of the agency and its frequently-evolving data. Data collocation between two Data Centres in Warsaw and Frankfurt offer high availability and improve the efficiency of the platform.

The usage of the microservices architecture allows for a relatively easy extension of computing power and appropriate management of replications, as well as helps to achieve the expected efficiency, depending on the complexity of the calculation of the product. Depending on the business needs and complexity of the insurance products, the expected performance is achieved by managing the number of replicas in the Kubernetes cluster.

The distributed architecture required a new monitoring tool – Observability. The solution providers (Crowtech & Zapasnik IT) decided to implement the tool based on ElasticStack (Elasticsearch, APM, Filebeat, Metricbeat). It offered an almost real-time access to log, trace logs and metrics that were ensured by fast hard discs and large bandwidth. The CI/DF pipeline has been introduced, developers from Crowtech published the changes and since then the system has been taking care of the distribution in the Kubernetes cluster.

Below is a high level Architecture

The benefits for the customer

The immediate benefit was a scalable andinnovative platform called PromesaOnline. The solution leveraged OVHcloud infrastructure and combined with Kubernetes orchestration, ensured both horizontal and vertical scalability.

The new process now fully-automated allows access to data for both the employees and Partners securely but efficiently . This system is available online for all branches and all Partners and allows for generation of customized reports.

The integration with Partners ERPs is now available, speeding the processes up immensely and generating considerable cost savings. Finally, automation allows for elimination of potential human errors.

It also generates new opportunities and integrates new larger customers – for instance, two large companies with urgent needs to renew insurance policies for their 20 000 vehicles. Integration of a new customer was reduced from 1 hour of manual data entry to zero. To give you an idea: with the old system in order to integrate 2 new customers, Promesa Plus would have had to employ 12 additional people for a year! Now it takes just the time to link both CRMs together.

The new solution also improved the employees engagement, liberating time for more creative tasks and leaving place for innovation.

Based on the experience of the solution providers (Crowtech & Zapasnik IT), they planned and implemented an enhanced disaster-recovery plan.

All data are cross-site replicated and stored in datacenters located in Europe.

With this new system, the company is now expanding its automated offering to other segments, moving back to the position of the innovation leader in the insurance market sector in Poland.

Why we chose OVHCloud?

1. Data Sovereignty and Security : OVHcloud ensures that the customer’s data is protected against possible interference, ISO 27001 certified DCs and compliant with European regulations thereby guaranteeing data sovereignty.

2. Predictable pricing : OVHcloud provides cloud services with simple billing and no hidden costs. No Ingress-egress Fees for customers wanting to move data around.

3. Best Price / Performance : Ovhcloud provides the best price/ performance ratio on the market and provide the platform to accelerate deployment of resource intensive applications and customer satisfaction.

Zapaśnik IT – Managed IT Services & IT Consulting, OVHCloud Advanced Partner

The Zapaśnik IT provides services to clients using Windows & Linux systems. In need of security assessment, security review, migration or new server setup? Call us, tell us what you need and we will surely find a suitable solution for you!

Crowtech – Software development and Cloud services

Crowtech is formed of goal-oriented professionals, who, thanks to their collective experience and skills, providing high-quality solutions and IT services for financial market.

At OVHcloud, we are proposing Object Storage in our Public Cloud universe and we have released, last year, compatibility with the S3 API, the market standard. Simply put, organizations and developers in particular, can reuse the tools (SDK, libraries or plug-in…) they are already familiar with, to migrate their data from any S3 bucket to OVHcloud Object Storage, and benefits from the cost-effectiveness, reversibility and privacy by design of our service.

This blog post details how Systran, a pioneer and global leader in translation solutions, has migrated from AWS S3 to OVHcloud Object Storage, and will cover the drivers behind that migration and the benefits. Huge thanks to Jean-Sellenart (CEO) and John-Paul Barraza (CIO), at Systran for helping us putting together this post.

Systran technology hosted at OVHcloud

Why does storage play a key role in Systran’s business? What are the reasons for migrating to OVHcloud Object Storage?

For Systran, storage is considered to be very important, as it hosts business-critical data such as translation models and models-related data. This is vital for the business in terms of Intellectual Property, as well as affecting the data transparency and privacy promise Systran makes to its trainers. As a result, trainers expect robust protection for the models they will list on Systran’s marketplace, as well as sole ownership over their data and reversibility; their biggest concern is that the models could be used for a different purpose than what they were designed for, which would lead to loss of earnings for them.

As a software solutions provider, Systran monitors its production costs closely, as they affect its margin. Systran was initially using AWS S3 as its service for storing models and related data, but they realised that the billing was unpredictable, too complex to understand and, most of all, expensive. As Systran’s marketplace gained more traction in the market, their storage volume needs were increasing. At the same time, it wanted to have more control over costs, while increasing its protection of its business-critical assets. As Systran was already an OVHcloud customer, it naturally chose to migrate its data to OVHcloud Object Storage to address those issues.

What advantages does Systran gain from migrating to OVHcloud Object Storage ?

The first resulting benefits are cost reduction and predictability. In fact, payback was achieved only two months after the migration. The total cost of ownership (TCO) with AWS S3 would have been around $13k for three months, while with OVHcloud Object Storage the TCO, including migration cost, was $11k for the equivalent time period. The benefits of migrating to OVHcloud from a financial perspective is becoming even more apparent by the month, as the bill with AWS S3 was around $4k per month, which is more than 250% higher than what Systran is paying for the same volume of data with OVHcloud Object Storage. Furthermore, as its storage volume is increasing by an average of 10TB every month, the pricing model of OVHcloud Object Storage is simple, predictable and transparent, which is critical for Systran in terms of better controlling its costs.

From a more functional standpoint, Systran was able to streamline its storage needs, as the migration was an opportunity to delete redundant translation models, as well as provide more protection and resilience to the data stored in Object Storage. While utilising the triple replication provided by OVHcloud Object Storage, Systran also decided to archive its translations in the OVHcloud Cloud Archive to increase its business resilience. Therefore, with OVHcloud, Systran can deliver on its promise of data privacy (thanks to OVHcloud’s GDPR compliance), reversibility and resilience for its ecosystem of partners, while protecting Intellectual Property.

Finally, Systran’s migration to OVHcloud Object Storage was facilitated by the provision of standard APIs. OVHcloud Object storage offers both S3 and Openstack Swift APIs which are widely adopted standards for development, meaning that developers can use tools that they are already familiar with. Another technical advantage of OVHcloud Object Storage is the network performance, which Systran has noted to be faster than AWS S3. This means that training translation models are faster for Systran trainers, resulting in further cost reductions.

Who is Systran?

Systran is a pioneer and leader in translation solutions, serving global organisations with its Machine Learning technology. With a key mission to provide tailored translation solutions to vertical industries, Systran has launched its marketplace: a community-based platform providing specialised translation models in various fields (such as IT, legal, and manufacturing), with full data transparency and privacy. This platform lets developers of specialised translation models (called “trainers” in Systran’s terminology) list their own model(s) on Systran’s marketplace, and end customers can buy those models for use on their own business applications or website. Systran collects the payment from the end customers, then pays trainers for the purchased models.

Towards the end of 2016, Systran was once again breaking new ground, launching the first professional neural translation engine. This program makes use of the latest advancements in deep learning technology to improve the quality of instant translation. If you want to know more about them, see this use case.

In this post, I explained how we had to push proxySQL past its limits.

Julien explained here how difficult it can be to hijack pieces of a software program, and make it work for a very specific use case. One so specific that we just could not push it upstream, as we were breaking some assumptions on the base code (something like, “look, we don’t use this, maybe we can divert it, to make it match our use”).

But sometimes, as we dig deeper and deeper through lines of code, pushing the limits of open-source software programs further and further, we reach bugs.

Here are two real-life case where we had an unexpected behaviour from ProxySQL, resulting in a patch for MariaDB/MySQL.

1. The importance of quoting

An unexpected behaviour

When we first started using ProxySQL, we had an unexpected behaviour. ProxySQL used to hang. No warning, no precursor. A hang means that ProxySQL cannot process your requests, causing your websites to be unavailable as a result.

Digging

Sysadmins know that when in doubt, check the logs. That’s what we did, and we noticed this:

[ERROR] Detected a broken connection during SET NAMES on 192.168.59.272 , 3306 : 2019, Can't initialize character set (null) (path: compiled_in)

So we took a closer look at the logs, tried to replicate the behaviour, and finally came across this:

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'binary' at line 1

This is clearly an error returned from the MySQL server. As this log line was generated on the ProxySQL, it means the error was between our ProxySQL and the MySQL server.

Finding gold

Julien worked hard — he read tonnes of logs, traced many PIDs, and tracked the problem down to THE triggering case — issuing this command:

set names binary COLLATE binary;

A collation is a kind of rule-set for comparing strings. It can define, for example when sorting alphabetically, if A comes before a, if é should be treated as e or not, or where œ should be in the alphabet.

You can read more about it on MariaDB’s Knowledge Base.

The fix

After submitting an issue on ProxySQL’s bug-tracker with what we discovered, and proposing a patch, the author of ProxySQL had a look and confirmed the bug.

While writing ProxySQL, René Cannaò did what all developers do — he followed MariaDB’s connector documentation. And the bug was from here:

According to the documentation on SET NAMES syntax (https://dev.mysql.com/doc/refman/5.7/en/set-names.html) :

charset_name and collation_name may be quoted or unquoted

This doesn't seem true for "SET NAMES binary COLLATE binary" , that requires the collation name to be quoted.

(https://bugs.mysql.com/bug.php?172id=93692)

The butterfly effect

So, from a hang on our infrastructure, we reported a bug to the project’s creator, who then traced it to a bug in a MariaDB connector, followed it up to its parent MySQL, and fixed it upstream. The bug was closed in early 2020.

In the meantime, we worked with Mr Cannaò to provide a workaround (basically forcing the collation name to be quoted in ProxySQL’s code).

2. When wires get crossed

While writing this article, I remembered another funny bug. I decided to check this bug status, and noticed we never reported it. My mistake is now fixed.

An unexpected behaviour

When using some real but exotic charsets on ProxySQL (we also love tricky scenarios, so we try to test them as much as possible), we had a ProxySQL error, stating we were using an incorrect charset.

MySQL_Session.cpp:2964:handler(): [WARNING] Error during query on (42,192.168.59.272,3306): 1267, Illegal mix of collations (utf8_general_ci,IMPLICIT) and (dec8_swedish_ci,COERCIBLE) for operation '='

Digging

Of course, first of all we checked that we were using a real and legit charset and associated collation. We double- and triple-checked that we should be allowed to use `dec8_swedish_ci`.

We decided to have a look at the connector’s source code.

Finding gold

If you are curious about it, you can have a look at older versions of the `libmariadb/my_charset.c` code, starting from line 541. You will notice that `dec8_swedish_ci` is nowhere to be found. But if you look closely, you will notice dec8_swedisch_ci. Dear friends from Sweden, you were not the only ones with a typo.

The fix

When fixing our issue with exotic charsets, we applied a custom patch on our custom build of ProxySQL. In the heat of the moment, we delayed the bug reporting.

We forked the mariadb-corporation / mariadb-connector-c GitHub repository, fixed some typos, and proposed a pull request — which was then merged mid-February.

The butterfly effect

Everyone is prone to typos, and making mistakes. We encountered some, they were fixed and ported for everyone.

Conclusion

As a leading web hosting provider in Europe, we face the Law of truly large numbers (https://en.wikipedia.org/wiki/Law_of_truly_large_numbers). TL;DR: Each and every event with a non null occurring probability will happen.

We host more than 1.3 million websites. Let’s assume that 0.0001% of our customers can trigger one single specific bug — we have at least 1 customer who will trigger it. The question is not if, but when we will have to deal with it, and how to do so.

Wondering how we noticed that we had hit the Law of truly large numbers? Stay tuned, because we will write about this soon.

Sharing is caring

OVHCloud loves open-source, and loves sharing. This is not the first time we contributed to the open-source world, and definitely not the last.

Life was not easy for it at first, but it soon learned how to handle the diverse range of configurations your websites have, particularly when it comes to making sure the path connecting to your data is always up.

Databases connection 101

Let’s go back to how a user connects to its MySQL/MariaDB database without a ProxySQL…

The user uses the MySQL client, and provides a host/IP, a username and a password.

From this point of view, everything seems pretty simple. But in real life, things aren’t so easy…

First of all, MySQL has evolved through the years to fix various security issues, such as removing their old authentication algorithm. Our infrastructure tries to keep up with all these changes, but there is always latency when migrating from the old versions to the newest version, especially as some of these changes can impact your website.

We are in luck though, as MySQL clients and servers have a good level of backward compatibility. 

So what do we have then?

Now we are closer to reality.

Now let’s add our little friend, ProxySQL, to this picture!

First hours of the life of a ProxySQL at OVHcloud

The ProxySQL is a very nice and useful piece of software. Its main job is to establish a connection to your database on your behalf. So a connection arrives, the ProxySQL looks up the database in its configuration, checks the credentials the user provided and, if they are indeed legit, act as a MySQL client to connect to the actual MySQL server.

From this point of view, everything seems pretty simple. But in real life, things aren’t so easy… Wait, did I already say that?

Let’s add our full diversity to this picture, just for fun! Remember our MySQL versions? Two specific versions were mentioned…

MySQL5.6

This is the latest version at OVHcloud. What is interesting with this specific version is that the authentication mechanism is the latest and greatest from MySQL. It generates a 41-byte hash, a change that was introduced in MySQL 4.1.

The actual version we had in P19 was MySQL5.5, but those two versions are pretty similar from a practical point of view.

MySQL5.1

Pretty old stuff, but we still have some instances using it.

Some customers using this version were migrated from MySQL4.0 and kept the passwords generated by that version. To keep this simple, we will refer to those as ‘MySQL5.1 with old passwords‘. This version uses a 16-byte hash, which is weak and prone to interceptions.

So what do we have then?

There we are! The journey of our ProxySQL does not go so smoothly, as it cannot authenticate against MySQL 4.0. This makes sense, as it was written long after support for MySQL 4.0 had been dropped.

Nonetheless, we needed to find a solution that took into account the legacy authentication and would prevent outages for the websites using those connections.

Under the hood: a MySQL connection story

As we highlighted earlier, to connect to a MySQL server, three parameters need to be provided:

• A host (say, mysqlXXX.sqlXXX)
• A user
• A password

The interesting part here is the password, as this is what changed between the two versions.

Obviously, MySQL does not store your password in plain text, but a hash instead. To enable ProxySQL to connect, the hash is inserted in its internal database, after which the connection is possible.

Three type of passwords exist:

• Plain text
• mysql_native_password (the 41-byte hash)
• old_password (the 16-byte hash)

ProxySQL only supports the first two… and therefore cannot connect to your awesome but obsolete database.

What are our resolution paths?

The simplest solution would be, as you can imagine, to update the passwords to use the new algorithm.

However, we do not store your passwords!

In some cases, we were able to compute this, but given the significant number of customers facing this issue (more than 50,000… remember the 1% of more than one million?), this was not doable.

The first step was to teach to our ProxySQL how to handle the connection from the client. Several patches later, our ProxySQL was proud of its new feature, and able to perform that task.

Let’s put it into a picture: 

But we definitely didn’t want to keep those old passwords forever… so we had to be nasty.

Remember, the only information we had was the legacy hashed password, which was the only source of information we can give to the ProxySQL. To simplify things, here is the situation we are in:

We needed to give our ProxySQL client a password, but could not add the old password (and didn’t want to!).

In the end, the solution was to take the old password hash and make a new password out of it. This way, we would have a new password, in the correct format!

As the ProxySQL client and MySQL server are hidden from the user, this was all fine, but we couldn’t do that in the ProxySQL server (you want to keep the same password, right?).

Lucky for us, we had the plain-text password field, which remained unused, so we could use it to store your OLD_HASH and complete the authentication chain.

We did it! You can finally access your database!

Let’s be clear here though… This is all very temporary, and as stated in the previous article, we do not want to maintain a forked version of ProxySQL where we can’t update the upstream code, as all this is very legacy-based and specific.

Conclusion

Maintaining a diverse range of software with incompatible versions is tough!

To make sure the migration had as little impact as possible we had to fix many small details. In the specific case detailed in this article, we had issues with old MySQL passwords and, given the timeline, could not ask MySQL5.1 users to change their passwords. But with a bit of hacking, we were able to start the migration… even for legacy authentication!

Again, this was only one of the issues we faced while setting up the ProxySQL and adapting it to our very specific requirements. Remember, tackling legacy software is a difficult and demanding task, but there is no avoiding it! This is what we did with the P19 migration, and what we will keep doing to always improve the service we offer you.

In this post, we will see how we managed to move all of them (almost) seamlessly…

The basics

How does a website connect to a database that’s hosted on a different server?
Short answer: by using the network.
As a diagram is always better than a long answer, here is a simplified view of how it works…

The webserver checks the website’s configuration to determine where the database is located:

$database_server = 'mydatabase.mysql.db';
$database_name = 'mydatabase';
$database_user = 'mydatabase';
$database_password = 'correct horse battery staple';
# ref: https://www.xkcd.com/936/

Then it starts its dialogue with the Domain Name Server (DNS), to obtain the IP behind mydatabase.mysql.db. Once the IP is known, the webserver can talk with the database server.

But having one network name for every database is something relatively new in OVH history. It was first introduced five years ago, with the Gravelines’ architecture described here.

A 20-year legacy

In the beginning, we used to give our customers the IP address of the database server.

On the positive side, you avoided querying the DNS server, you saved time, you avoided a possible bottleneck, and you may have even avoided a potential point of failure.
On the negative side, if we had to change the server and its IP couldn’t be moved – let’s say, when a migration occurred – every customer had to manually change their website’s configuration.

$database_server = '10.0.59.42';
$database_name = 'mydatabase';
$database_user = 'mydatabase';
$database_password = 'correct horse battery staple';
# ref: https://www.xkcd.com/936/

As we have loyal customers (including many of you, in this case!), it was not a bad idea, but not really scalable.

So, to ease server management, hardware end of life etc., OVHcloud switched to providing our customers with the server’s name instead of the server’s IP.

$database_server = 'mysql55-59.pro';
$database_name = 'mydatabase';
$database_user = 'mydatabase';
$database_password = 'Brussels Sprouts Mandela Effect';
# ref: https://www.xkcd.com/2241/

To sum-up, we have three ways to connect to a database in P19:

• Using the server’s IP
• Using the server’s name
• Using the database’s name

The blocker

Following the rules described here, as we moved your databases individually and not a whole server at a time, and databases were shuffled across all our new instances rather than staying with their neighbour, we could not reuse the server’s IP address. Also, we could not reuse the server’s name.

We seriously thought about automatically changing all our customers’ configurations.

Be this meant we would have had to:

• Parse hundreds of terabytes. Doable, but time-consuming.
• Understand your website’s organisation. This can be automated for up to 99% of all different uses cases, but 1% of 1.5million websites is still 15.000 potentially broken websites following the replacement.
• Replace the IP or servername with the database name. Possible, but not 100% reliable.

What if the customer:

• Uses its database in a binary file, like a compiled CGI written in C? Not parsable.
• Uses a different database than the one we have the link? How would we know what should be used?
• Does not use its database? How would we know it was actually the correct result if we found nothing?

As this was clearly not an option, we had to find a way to support every use case.

If the configuration was:

• mydatabase.mysql.db: No problem. The DNS was up-to date.
• servername: The query would reach the server and need to be forwarded.
• server IP: The query would reach the server and need to be forwarded.

The solution

Maybe some of you have already started to figure out the solution…

We needed a piece of software to masquerade as the server, but actually redirecting incoming connections to the real server behind it. In other words, a proxy!

We decided to use René Cannaò’s proxySQL.

ProxySQL is more or less designed for a small number of users, against a small number of databases, mainly for splitting read-only and read-write access on different backends, which can greatly help a database with few writes and a lot of reads support the load. It can also help parsing and securing queries. But using it for our purposes was pushing it to its limits. We must say, this product rocks!

Paris side

First of all, we had to deploy it on hundreds of servers in P19.

Our architecture allowed us to install one ProxySQL on each database host. This was done between the 27th November and the 6th December 2019 (http://travaux.ovh.net/?do=details&id=35499).

Once all of them were up and running, we could start the migration. For every database, the ProxySQL was updated, so it would track where the database really was, in Paris or in Gravelines.

Easy-peezy, isn’t it? As the ProxySQL runs on the same host, we didn’t need to fake anything and it worked like a charm. But what would happen if a webserver in Gravelines called a database that was still in Paris?

Gravelines side

This time, we did not have the choice but fake the server name in Gravelines. What could we use? A ProxySQL of course! But on our new architecture we just could not install them on every hosts, because as seen previously, we have 10 times less customers per instance in Gravelines than in Paris.

So we had to set up a ProxySQL farm and a lying DNS which resolves to a member of the farm for queries about servers in Paris. And using the same mechanism for updating ProxySQL configuration during migration, we finally had this kind of infrastructure:

But what about configurations with IP addresses in them? As we had ProxySQL loaded with users for a group of P19 servers, we could – thanks to a routing trick – make the webservers think that P19 IPs were set up on a ProxySQL!

The downside

You just learned that to allow the migration to happen in the required timeframe, we had to introduce ProxySQL. Even if it is a great piece of software, and was almost a life-saver for us, as sysadmins, we prefer to avoid maintaining more and more different solutions on a daily basis. This is why we have decided to drop support for ways to connect to your database other than .mysql.db in the months to come. More details will come soon.

If you want to retain our state-of-the-art way of doing things (and also give us a hand with keeping the changes smooth!), you can check that your website already uses .mysql.db in your config files.

Conclusion

Having to support multiple ways of connecting to a database gave us loads of headaches. Finding a great piece of software was our aspirin, and allowed us to enable the –quiet flag in our migration process, so customers did not have to change their configurations for their websites to stay online.

However, you may have noticed it did not go quite as smoothly as we wanted. We encountered many small, sometimes unexpected, problems that we had to fix ASAP…

You may remember the abnormal amount of “max_user_connections” (http://travaux.ovh.net/?do=details&id=35689).
We had to play with ProxySQL’s source code to enable its use against mysql 5.1 combined with the pre-mysql 4.0 password hash algorithm.
We found funny things about encoding in the source code we patched upstream.
We were able to flood our host’s ARP table multiple times per hour.

You will learn more about these challenges in our upcoming posts!

The rules

• To reduce latency, a database must be migrated at the same time as the website using it.
• Because databases are spread across all available MySQL servers, the granularity of a migration has to be the database, not the MySQL instance. In other words, we cannot migrate a whole MySQL server. We must only move a part of it.
• As the link between a website and its database is not necessarily referenced, a website in Gravelines should be able to contact a database in Paris (for example), and vice-versa.
• To contact its database, a website uses a host name, a user name and a password. We want the migration to be transparent, so nobody should have to change any of those elements to contact their new database.
• Databases’ platforms change between Paris and Gravelines, as shown below.

To sum up, this is what we have before a web cluster migration:

And this is what we want after the migration:

A few more things…

• Obviously, we had to keep in mind one of the most important thing when dealing with databases: consistency. For each database, we had to define a consistency point. Before this point on the timeline, read/writes were made in Paris. After it, read/writes were made in Gravelines.
• We believe in transparency and reversability. These are both key parts of our SMART cloud. That’s why we wanted to give you access to this consistency point, as a dump on your OVHcloud Control Panel. For every database migrated, we chose to give you access to the dump for one month.
• Migrating 800K databases in about 60 nights meant we had to be very fast and scalable. Our record was on the 1st of July 2019, when we successfully migrating 13,502 databases in 1 hour, 13 minutes and 31 seconds.
• If you’re used to being on duty, you know that your attention and efficiency are lower during the night. Repeating the migration process about 60 times in one year would amplify that, so we wanted everything to be as automated and as simple as possible. As we will see later, to launch the database migrations, we just had to run one command on a single host:

migrate-p19

Now you know rules, it’s time to start the game!

Level 1

The first level is always an easy one, where you discover how the game works through a kind of tutorial! So, let’s begin with a small database migration. This is how we do it:

1. At the source (Paris)

• Set read-only mode. We absolutely need to avoid writes during the migration, to avoid the famous split-brain. The easiest way to do this is to put the database in read-only mode. In most cases, websites need only to read the databases, but in some cases, they need read and writes, and they will therefore be broken. That’s not a problem, because the website is currently migrated and is closed. We’ll block write access, in case the database is used by another host that isn’t affected by the nighttime migration.
• Dump the database and put the dump somewhere. We choose to store the dumps on OVHcloud’s Public Cloud Storage (PCS), as we already use this solution to store 36 million dumps per month. Adding 800,000 dumps in one year is no problem for this awesome platform!

2. At the destination (Gravelines)

• Retrieve the dump and import it.
• Create the user and permissions, with write access.

3. Switch to the new database

• At this point, the website is still calling the database at Paris. So the website (whether it’s hosted in Paris or Gravelines) can contact the new database, we’ll update the DNS so that the name points to the Gravelines MySQL instance rather than the Paris one.
• The read access to the Paris database is also removed.
• Finally, we’ll update our information system, so you can retrieve the dump from PCS via your Control Panel. This update also allows us to redirect all actions available from the Control Panel (e.g. change a password, create a dump…) to the new database at Gravelines.

Level 2: “Decentralised state machine”

To deliver the proof of concept for the migration, we first ran all those steps manually and sequentially. The natural way to automate this is to write a script to do the same thing, but faster. That’s a centralised method, but such methods experience bottlenecks sooner or later, and imply a single point of failure.

To prevent this, and fulfil our scalability requirements, we have to be decentralised. We must imagine the migration of a single database as a state machine. Here is a simplified version of the state graph of a database migration, as described above:

Using this state machine, we can perform those three big steps by different machines to parallelise the workload:

• The source
• The destination
• The one updating the DNS

These three hosts can perform their tasks in an independent and decentralised way. All they have to do is watch the state graph to see if they have something to do, and if so, update it and perform the tasks.

The brain of the migration: CloudDB

We love the ‘eat your own food’ concept! That’s the best quality control ever, and with the feedback you give us, our first source of feature requests. So it’s no surprise that we used our own CloudDB product to store the state graphs of the databases migrations.

Technically, a state graph is a row in a table. The simplified structure of this table looks like this:

- database_name VARCHAR(255) PRIMARY KEY,
- source VARCHAR(255),
- destination VARCHAR(255),
- status VARCHAR(255) NOT NULL DEFAULT 'Waiting',
- dump_url TEXT

Except for the dump_url, all the fields are filled before the migration begins. In other words, we know where the databases are, and where they will be.

We’ve beaten all the challenges of this level. Now it’s time to beat the final monster!

Level 3: Migrate 800K databases

Now that we know how to migrate a single database in a decentralised way, let’s fill the CloudDB with all the databases we want to migrate! This is what the migration now looks like:

In Paris

About once a minute*, each host of the 780 database servers asks the CloudDB if they have something to dump. The source and status columns of the table are used to retrieve this information:

SELECT … WHERE source = me AND status = 'To dump';

If so, they execute their tasks and update the CloudDB about what they’re doing. When they’re done, they pass the baton for this migration to Gravelines:

UPDATE … SET status = 'To import' WHERE database_name = '…';

In Gravelines

At the same time, 300 kilometres away, hundreds of database servers are also asking the CloudDB if they have something to import. As in Paris, they query the CloudDB about once a minute*. The destination and status columns of the table are used to retrieve this information:

SELECT … WHERE destination = me AND status = 'To import';

If so, they execute their tasks and update the CloudDB about what they’re doing. When they’re done, they pass the baton to a third robot, who will change the DNS records for this database migration:

UPDATE … SET status = 'DNS to update' WHERE database_name = '…';

(*) To avoid flooding the CloudDB, we use a random frequency to query the database with the state graphs. This way, the connections are globally distributed over time.

DNS update

The robot responsible for the DNS update is the third player in the migration process, and works in the same way as the dump and import robots described above.

Not that easy…

Of course, the actual game was more complex. This was a simplified version of the migration, with some steps missing or not detailed enough, such as:

• Preventing writes to the source database
• Updating the IS (among others) so you can see the dump in the Control Panel
• Setting the password on the destination (the same as the one on the source) without knowing it
• And many others

But now that you’ve got the concept for the main steps, you can imagine how we handled the others.

Cheat code: Iterate!

Do you know the law of truly large numbers? It states that with a large enough number of samples, any unlikely thing is likely to be observed.

This is one of the first lessons you learn when you host 1.2 million databases. Every day, we face the many improbable things that can happen with databases, so we knew that, despite the tests we conducted, we would encounter difficulties, weird cases, and improbable bottlenecks.

But there’s a cheat code to beat that boss: iterate!

• Begin the migration
• Face a problem
• Fix it definitively (not just for the particular case that failed, but also for all similar cases across the whole platform)
• Then try again, faster!

This method is possible thanks to two things:

• The magic command
• The big red button

The magic command

As mentioned above, to launch the database migrations, we had to run one command on a single host:

migrate-p19

This magic command has one parameter: the number of parallel migrations you want to do. We used 400 for this parameter.

migrate-p19 --max-procs 400

This means that 400 databases are being dumped or imported at the same time –  no more, no less.

The migrate-p19 command is a scheduler. It updates the CloudDB every 10 seconds, so we always have those 400 migrations performed in parallel:

SELECT COUNT(database_name) … WHERE status in ('To dump', 'Dumping', 'Dump failed', 'To import', …);
42
UPDATE … SET status = 'To dump' WHERE status = 'Waiting' LIMIT (400 - 42);

Pause the game: The big red button

In every machine, it is mandatory to have a big red button to press when something’s wrong. To interrupt the migration, for a reason or another, we just have to kill the migration-p19 script. When we do this, the ongoing migrations terminate themselves, after which no new ones are launched.

The interruption is automatic. If something’s really bad, the scheduler can see that there are too many operations in an error state, and then decide to stop the migration.

Our database migration adventure will continue on this blog. Indeed, the world is not perfect, and we’ve had to handle some interesting cases…

To be continued…

Want to be part of the team bringing this migration to life, and keeping this huge platform in an operational state? We are hiring an open-source SRE and automation addict in France and Canada!

In these new series of posts, we will tell you about our experience with web hosting databases.

How to handle 800k databases?

Our previous posts explained why we have to migrate databases along with their websites. You also know that we set up a network bridge for customers using non-referenced databases.

Yet migrating databases also brings challenges. We will describe the most important ones.  But first, a little reminder of our different architectures.

OVH offers two types of databases:

• Shared databases (which we call SharedSQL)

• Private databases (which we call, you guessed it, PrivateSQL)
  

What is a SharedSQL?

As the name suggests, it is a database on a shared server, which contains many other bases, and which shares its hardware resources, including RAM.

Of course, not everyone has access to everyone’s data! Each database is defined with a single user able to access it.

This technique allows us to exploit up to 2500 databases on a single server. This is much cheaper than offering a physical server per customer. This is how we can offer low cost access to several databases within our hosting offers.

What is a PrivateSQL?

Unlike SharedSQL, PrivateSQL are databases with guaranteed resources, including RAM. In PrivateSQL, users also have many more rights and greater customisation capabilities.

Does each user have their own server? Not really! Several years ago we used Docker technology to containerise our databases, we already discussed it in this post: https://www.ovh.com/en/blog/docker-administration-databases-a-flying-ideas/. With a PrivateSQL, it’s not just the database space that is private, the RAM assigned to the service is also guaranteed. This means that in no matter the circumstance, the performance is consistent.

Seven differences!

When considering migration, we had to study the difference in architectures between Paris and Gravelines.

Regarding PrivateSQL, it was simple: there is no difference, we had already aligned the architectures when we containerised them. Migrating them would be a piece of cake: we just had to move a container from point A to point B.

At least on paper, in real life it was not so simple, but we will talk about it next time.

Here is a simplified diagram of a PrivateSQL

In contrast, SharedSQL databases were not really aligned. When we set up the Gravelines datacentre in 2016, we took advantage of Docker’s flexibility to review our database technology, and therefore adapt our former solution.

Small comparative schema of a SharedSQL in Paris and Gravelines:

SharedSQL at Paris P19 and Gravelines:

On the one hand (in Paris), we have servers with a single database management system (MySQL) hosting 2500 databases on one machine.

On the other (in Gravelines), we have added a level: Docker containers have the same database system (MySQL) hosting up to 250 databases. On each machine, we host 10 containers.

This new system greatly simplifies maintenance, reducing the impact on database performance as it occurs.

Here is what it provides:

	Paris	Gravelines	Explanation
Number of clients impacted by an incident on an instance of MySQL	2500	250	When MySQL is on a container, the impact is lower than when it is directly on the server.
Total number of customers per server			Equivalent. In Paris, the 2500 databases operate on the same MySQL process. In Gravelines, they share 10 processes, each in their container. But the number per physical machine remains the same.
Easy server migration in the event of a hardware failure			Docker containers provide us with ease of migration and disaster recovery between two machines. Thus, we can easily and quickly move a container from one server to another without data loss.
Startup Speed			Starting a container is about the same as starting a process.
Version upgrade			The transition from one version to another is simplified with the Docker image system. We just need to change the image of the container to the desired version, and restart the container.
Rollback in case of problem during an update			We just have to restart the container on its previous image.
Robust and long-lasting implementation			Running databases under Docker is not simple (see https://www.ovh.com/blog/docker-administrating-databases-a- crazy-idea /), but we have a few years’ experience.

Of course, running a database in a Docker container seemed like a crazy idea. But after many years of production, this technology simplifies everyday life in the Gravelines datacentre.

So when we had to choose between replicating the Paris infrastructure, or changing infrastructure during migration, we chose to change technology.

And all of a sudden we have another migration challenge.

In our next article, we will detail how to move a huge database in a few minutes, while changing a large part of the software and the infrastructure behind it!