The flexibility of the NoSQL data model

Unlike traditional databases, MongoDB can gather data without a predefined structure and adapt its dynamic scheme as needed. The tabular schema is no longer used, as regards NoSQL databases, the logical representation of a NoSQL database resembles a tree structure. MongoDB is a document-oriented database management system that organises data in JSON documents and collections. These documents contain pairs of values or keys of different types, including nested documents and arrays. The structure of key/value pairs can vary across different documents. Additionally, MongoDB also allows for the use of GridFS, which enables it to function as a file system using the JavaScript query language.

Advantages – based on use cases

MongoDB and MySQL both have their various strengths and benefits, which include indexing, query language (both rely on natural language and offer a graphical interface when needed), security (each offers authentication, access control and data encryption), and compatibility with various programming languages (Java, Python, Node.js, server-side PHP, Ruby and C#). Scalability and strong read-write performance set MongoDB apart in multiple use cases.

MySQL only allows for vertical scaling, i.e., you can only add resources to the database server (RAM, CPU) for scalability. During the replication process, read-only copies are created on other machines, which may result in delays in updates and the user’s access to information.

MongoDB goes even further by incorporating both vertical and horizontal scaling into its design. Two mechanisms, clustering and sharding, are used to achieve this. Clustering, which involves groups of MongoDB servers with identical data (three servers minimum), and sharding, which spreads different parts of the data across multiple servers for load distribution, high availability, and security in case of hardware failure.

MongoDB stands out from the rest in terms of performance, particularly in writing, when using the “insertMany()” API command to insert or update numerous records. When dealing with a high volume of records, MySQL is the faster option.

So, your project, available hardware, and the type of data you work with (structured, semi-structured or unstructured) will determine the most suitable technology.

Of course, there are alternatives or plugins to address the limitations of MySQL or make up for missing features – depending on the use case. For instance, with an RDBMS like MariaDB, you can enable multi-master clustering using this approach. However, attempting to adapt a technology to needs it wasn’t designed for in the first place will inevitably make maintenance complex, increase the risk of malfunctions, and lower performance.

On the other hand, if MongoDB proves to be more suitable, we can’t overlook the skills needed to deploy, configure and maintain MongoDB’s architecture, as well as understand replicate set and sharding ^[1] (Find out more about the differences between the two). If you need it, choosing an OVHcloud-managed solution like Managed Databases for MongoDB could be the ideal option. 

Why choose MongoDB over any other non-relational database management system?

There are many other document-oriented databases out there, including Couchbase, CouchDB, Oracle NoSQL Database, Amazon DocumentDB, Azure Cosmos DB, and more. Nonetheless, MongoDB is notable for its versatility, performance, scalability, and features. This makes it a popular DBMS for various use cases, including e-commerce catalogues, web/mobile apps with user-specific content customisation (e.g., social media), payment systems, serverless design, workload migration, content management, gaming, IoT data collection, and much more. It spans use cases involving big data and artificial intelligence.

Unlock MongoDB’s full potential effortlessly with OVHcloud

Installing MongoDB on a single server, VPS or Public Cloud instance is straightforward. However, things might get more complex when the project enters production and/or gains momentum, or requires a stable environment and deployment of a multi-node cluster. This also includes mobilising specialised resources that could be allocated to your main business, like developing applications. This particularly holds true for use cases that demand solid compliance, such as health data hosting and management.

OVHcloud offers a managed MongoDB solution — i.e., it is managed for you by our team and hosted in a sovereign cloud. — designed and delivered in partnership with MongoDB’s teams. Since our Database as a Service (DBaaS) is not shared, it allows you to free up your SysAdmins without compromising performance and compliance. Each user has their own environment, except for the Free Tier Discovery offer, where CPU and RAM resources are shared. This service also helps you tackle the challenge of scaling your database, syncing and distributing data across several nodes, as well as backing it up (included in our plans).

Find out why you should choose MongoDB managed on OVHcloud, an expert in MongoDB infrastructure – including deployment and administration. You can achieve greater efficiency and build more reliable, powerful and secure applications, while optimising the total cost of ownership (TCO) of your data.

Want to go further ?

At first glance, the idea might seem absurd: migrating to NoSQL, a non-relational database management system, while using a MySQL or Oracle database. It’s like putting petrol in a diesel car. Of course, the code will need be altered, but it is certainly doable. Read the MongoDB webinar “Migrating relational database workloads to the MongoDB Document Model”

^[1] Soon, the OVHcloud Managed Databases for MongoDB service will include the sharding set-up feature. This has been added to our roadmap.

What’s new?

Over the past months, we continued to gather feedback from our customers and improve our offers. Now powering thousands of databases clusters, we focused on the major requests discussed directly with our customers, or from official Github roadmap and Discord community. From better flexibility to higher resiliency, let’s have a look about new features.

Additional storage space for flexibility

The first major improvement is about storage and flexibility. Before, each databases service was launched with a fixed amount of storage, depending of the nodes sizing. As a example, nodes with 16GB of RAM came with 400GB of storage. A recurring request of more flexibility, at creation but also once it’s running is now live with additional storage option.

During cluster creation, you can now opt for more usable storage space for your service. And once your cluster is up and running, you are free to increase this value without any downtime. So far you cannot decrease the storage space, but it’s prioritized for the incoming months.

You can directly verify minimum and maximum storage space from our website or control panel.

Secondary backup for the same price

We now provide a secondary backup location per service for the same price, for all Databases services except MongoDB (only one remote backup for now). One in the same physical location, and another one in a remote location (offsite backup).

Another recurring request is to provide a better fault-tolerance, for any situation, such as a rack failure, a datacenter failure or anything else. Since the beginning, we provide daily backups in the same geographical location. Being in the same location is very useful in case of localized issues. You avoid network latency, and improve restoration time. But remote backups were awaited. it’s now provided for free for any cluster, running or soon-to-be created. You can find your backup location directly in our API, and soon via OVHcloud Control Panel. More information in our official documentation!

Exact time machine with PITR

Point-in-time-recovery (PITR) is now available for all MySQL, PostgreSQL and MongoDB Enterprise services, directly via OVHcloud Control Panel or API. What does is bring?

One of the nicest feature from MySQL, PostgreSQL and MongoDB is Point-in-time-recovery, known as PITR. Imagine being able to travel back in your database operations, and restore your data at a really specific timestamp, such as “two days ago, 15h02 and 24 seconds” ?

PITR proved its efficiency over the years and saved the life of many databases admins. Imagine yourself deleting the wrong table at 15h02, and being able to restore the minute before instead of the backup from last day? This is not black magic, but only a combination of daily backups and logs files (such as WAL logs for PostgreSQL).

… And much more !

In case you missed it, we also released tons of improvements in the last six months, here we go !

• Latest DBMS version with PostgreSQL 15 and MongoDB 6.
• Automatic email alerts when your cluster is reaching storage limits (also, you can retrieve the disk usage with our API)
• Ability to terminate PostgreSQL/MySQL queries directly from control panel or API. Quite useful when a specific query is impacting your cluster.
• Much more control over MongoDB roles from control panel and API. More information.
• More storage by default for MongoDB offers. Don’t hesitate to check our website!

What’s next?

Our roadmap for the next months is public, and still connected to customer requests.

The first item is about storage performance. While our clusters fit globally our customer workloads, advanced ones can feel performance limitation on the storage part. We are currently implementing better storage, reaching 20’000 IOPS instead of 3’000. It’s already implemented for the same price in MongoDB cluster, and soon for everything else (roadmap card).

Second major item will be a better integration with Kubernetes clusters, located in OVHcloud or any other provider. Today, you can link them together, but it will be improved with a Kubernetes operator (roadmap card), simplifying the authentication between them.

That’s all for today ! If you have specific needs of just want to give it a try, don’t forget that all the services are billed per hourly, meaning that you can test your use case for a couple of euros. and please join our Discord community to share your feedback after !

Hence, the start of this Public Cloud changelog. We’ll communicate regularly about the latest features to update our customers. For the sake of consistency, we will organize the section of this post, in the same way our roadmap on Github is organized. We hope you’ll enjoy it and find it useful. Don’t hesitate to share your feedback!

Infrastructure: Compute, Storage and Network

• Our entire OpenStack infrastructure has been updated from Newton to Stein version. Beyond versioning, the support of OpenStack Stein brings a host of improvements that will enable us to deliver new capabilities. For example, on the networking side, you can now leverage Security groups to manage network access rules for instances. More is coming soon on the network, so stay tuned on this.
• On the Storage front, in case you have missed our announcement, our Standard Object Storage – S3 API is now available in Gravelines and will be rolled out our other datacenters next. Providing complete S3 API coverage, you get the best price on the market, for storing any static file with unlimited scalability.
• This release complements our Object Storage portfolio, where High Performance Object Storage – S3 API is available in our Beauharnois datacenter. For reference, you can check the available list of services in our datacenters here.
• Last but not least, Object Storage – S3 API now supports server side encryption and S3 object lock, thus bringing more advanced security features.

Databases

We now offer more than 12 Database Management Systems, available as a managed service and billed upon usage within our Public Cloud. The key highlights from the latest months include:

• For all DBMS we do provide Terraform integration, easier connections with code examples and Docker and backup and maintenance timeslot adjustment. Check the video below for more details.

• For all DBMS, we have improved order ergonomy, so you can more intuitively and quickly select the sizing and features of your managed database cluster.
• We have added new flavors to all our DBMS catalog, now starting from 4GB RAM per node
• For MongoDB DBMS, we provide an easier way to connect with DNS Seedlist features

And as a video is worth 1000 words so have a look at the control panel to see those new features in action.

Containers & Orchestration

A growing numbers of our customers are now adopting a cloud native approach for their applications. Most of the new features we have developed over the past months are related to our Managed Kubernetes Service. Below are details about the purpose of each new feature.

Security

• Clusters spawned in vRack (OVHcloud cross-region private network) can now specify a custom Gateway IP (or use the one declared through OpenStack private network DHCP) to route all worker nodes’ traffic.

Automation & Scalability

• CoreDNS configuration can now be supercharged with custom entries that will be preserved during a cluster patch or upgrade.
• Following user feedback, we made the DesiredNodes property of Node Pools optional.
• We improved our Terraform Provider, assuring that Kubernetes resources cover our entire feature set.
• You can now choose the value of AlwaysPullImages

Kubernetes Lifecycle

• We released Kubernetes 1.24 for both new and upgrade to existing clusters.

Other key updates

• OVHcloud obtained KCSP (Kubernetes Certified Service Provider), meaning that with multiple CNCF certified experts, our Professional Services team can help enterprise customers in their cloud native journey with OVHcloud.
• We published new tutorials on policy management with Kyverno  and  Labels & Taint on Node Pools and updated more than 20 Kubernetes documentations pages.
• Next month, we will make a small breaking change to the NodePool CRD, giving you more precision when a NodePool is being resized.

AI

Our AI development team keeps on rocking and delivered many great features, including:

• What you used to know as AI Apps, has been renamed to AI Deploy and we released an Alpha version release.
• We are now offering on AI Notebooks a framework that can run quantum algorithms
• New python, open source libraries for machine learning, computer vision and image processing are proposed by default in AI Notebooks
• On AI Tools, we have improved rights management with “Read Write Delete” granularity
• Stand-alone volumes are available on AI Tools, so the experience is improved with framework that are using a lot of temporary files.
• We have released many new tutorials to make it easier to use AI Tools. Check them out here.

Too Long Don’t Read ? View those updates in video demo.

We can compare it to a game with different rule sets, on which every player must agree before playing.

The UNO analogy

Let’s make an analogy with the UNO card game. There is many variants, but we can sum them up like this:

Before starting to play, one defines uno_mode=DOUBLE,INTERCEPTION, which means the STRAIGHT_FLUSH rule does not apply. If this rule is activated later in the game, it may cause problems around the table, some players may have been disadvantaged in the previous rounds and may not want to play any more.

A card game to manage data?

SQL mode works like this, but instead of specifying how to play cards, it specifies what to do in certain situations:

• Is “2020-11-00” a valid date (NO_ZERO_IN_DATE)? It changes data validation performed by the DBMS.
• Is SELECT name FROM users GROUP BY first_name a valid query (ONLY_FULL_GROUP_BY)? It changes the allowed syntax.

When MariaDB 10.2 and MySQL 5.7 were released, the rules have changed:

We can note that newer versions kept old rules, but new ones were added. That’s why when upgrading from MariaDB 10.1 to MariaDB 10.2 for instance, one must specify to its database to play with old rules, and not to enforce STRICT_TRANS_TABLES, ERROR_FOR_DIVISION_BY_ZERO.

Concretely

Actually, if my website should store the result of a division, but the denominator is `0`, my database with the MariaDB 10.1 default SQL mode database will allow it and store the result as NULL. My database with the MariaDB 10.2 default SQL mode will also allow and store a NULL value it but will raise a warning. If strict mode is also enabled, MariaDB 10.2 will throw an error and store nothing. Thus, changing SQL mode can break websites and should be done carefully.

As we cannot check every line of code of every website of each of our customers, when an DBMS major upgrade changing the default SQL mode is planned, OVHcloud activates the legacy SQL mode ensuring your website will not be affected. Of course, you can switch to the new default SQL Mode in the “Configuration” tab of your CloudDB. But keep in mind this is for experienced users and OVHcloud advises to stick with the default sql_mode, unless your database was upgraded from a previous version with a different default sql_mode.

Official Documentation:
MariaDB: https://mariadb.com/kb/en/sql-mode/
MySQL: https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html

Aiven powers OVHcloud’s growing DBaaS portfolio, Managed Cloud Databases. Read this blog post from Aiven CTO, Heikki Nousiainen, to find out how their customers have achieved better agility and efficiency in the cloud – and how you can do the same!

Why business agility is important

For companies in today’s changing world, it’s crucial to remain agile and adapt to changing circumstances. Failure to change directions, sometimes in mid-step, can result in needless expense and hurt a company’s image. 

When we talk about agility, we often think of product and corporate strategy and other long-term progressive endeavors. That’s valid, since both the product roadmap and the overall strategy of the company must be able to change to fit the market environment, as well as in response to internal developments.

But agility is also relevant in operations and infrastructure, where a company’s internal workings must also be able to accommodate change. New innovations come to market every day that can save money in your operating expenses; and every now and then tools become obsolete, even though you’re still using them. If a company doesn’t react in a timely manner, these changes can result in reduced efficiency or actual loss of business.

What’s more, any company’s operations and infrastructure absolutely must support innovation on a fast cycle, or compromise their ability to adjust to change.

DBaaS in the cloud makes sense

One of the pieces of infrastructure that it most makes sense to purchase as a service is databases and related services. Setting up and managing a database is a specialized skill, especially doing it well, and requires resources that can instead go into developing a company’s own products. In practice that means that instead of hiring a database admin, you can hire a developer for your product team, while another company with its own skilled team is taking care of your data, and you can transfer that slice of budget from OpEx to development investments.

On a more general level, data is one of a company’s most valuable assets, and plays a key role in all modern online and software products. This means not just data as code and images, and not even the data that flows when the service or product is being used. Storing and analyzing customer and usage data can tell companies how their business is doing and how they should be directing future development efforts. Building the capacity for this data storage is an investment that’s hard to scope out in advance, and analysis needs will evolve as the business grows and the market changes.

Open source for freedom

Things change all the time, and whatever service you buy may not be around for as long as your company, or they may change their terms without consulting you. Or you might just outgrow their offering and need a new provider. 

When buying your data infrastructure as a service, the one thing you want to ensure is that you’ll always have access to your data, even when you decide to switch services. The data also needs to be in a format that’s compatible with other providers and products, or you stand to foot a massive bill for conversion (if it’s even possible).

Here’s where open source is key. If you go with a service that offers fully open source products, your data will always be in a format that’s easy to export elsewhere, and it’ll be compatible with a wide variety of solutions. Open source gives you real freedom and flexibility to choose your providers and not be locked into a solution offered by a single company.

Shortening the innovation-development-testing cycle

Managed cloud databases are increasingly the key factor that enables quick innovation. Product development teams need a way to quickly turn ideas into testable products, and ready-made, managed infrastructure is the only way to achieve this in the IT world. The more building blocks you can buy ready-made, the quicker you can sketch out a test version of your dreams, and the quicker the roll-out when it’s finished.

DBaaS customers can provide data-driven services without having to build up the expertise in managing a new system and then having to build a datacenter. Their developers can quickly get to the part where they start programming, which greatly shortens the time to market. Being ahead of the curve by months can make or break a product, no matter how high its quality otherwise.

And speaking of quality–having the means to spin up data infrastructure quickly also means a company can continuously improve their products. With a DBaaS-sourced architecture, new environments can easily be set up for testing. 

This is equally true when the issue is not just improving upon or enlarging existing services, but creating entirely new products. The road from idea to product is much shorter when the  infrastructure is already built.

And lastly, it also means many teams can work on the same codebase simultaneously. This way the company can make every R&D penny count.

From CapEx to reduced OpEx

Then there’s the fact that datacenters cost money to build, and you’re using your CapEx to do it: it’s an investment in the most classical sense. Even if you’re leasing server space instead of building walls, you’re still spending your OpEx coins in a predefined amount of capacity and its maintenance. 

The amount of capacity a company maintains has to accommodate not just everyday use but also traffic spikes. When an investment is made (usually in some hurry) to increase capacity in response to an emergency need, it’s seldom considered safe to let go of it when the emergency passes because the process is painful and slow. This road usually ends with services overprovisioned by some 30-50% for regular use. And this is true whether you’re doing it with CapEx or OpEx.
CapEx reduction is easy to achieve by moving to a managed service, and there are plenty of them. But by selecting the right DBaaS, a company can make sure that their capacity meets their needs on a day-to-day basis: any service worth its salt will make it easy not just to scale up but also to scale down. It’s possible to reach a 70% reduction in infrastructure OpEx by managing your capacity and infrastructure properly. At the same time it lowers the threshold for using these services for new products, as well as retiring products that aren’t generating significant revenue.

Orchestration as a multiplier

There’s an African proverb: If you want to go fast, go alone; but if you want to go far, go together. Creating a standalone product to serve a single purpose is quick, but to provide real long-time value for your customers and your own company, versatile products or services that offer a range of functionalities and meet multiple needs are key. You might hit a goldmine by creating a single-purpose product or service that satisfies an urgent user need that no one else in the world has thought to satisfy, or you do it better than absolutely everyone else. It’s just not very likely, to say the least. 

A more robust business model is to target a wider customer base and meet a variety of needs. Versatile products are more useful to a larger group of people, and customers are more likely to want a solution that meets several needs at once. Even if you start small with a tightly targeted product or service, you want to keep the door open to expansion and innovation available in adjoining spaces.

The reason that products like this are harder to create and hence more valuable is that it means creating a heterogeneous data architecture that encompasses a wide range of building blocks. This is not easy, whether you’re building them in-house or buying them as a service. 

And making the components in your infrastructure all work together and pass data along flexibly is actually really hard. Companies that buy their data infrastructure as a service may have it even harder: using several services from multiple providers can add to the complexity of the overall architecture, and the range of available services can limit what it’s possible to accomplish. 

Note that the data flow itself is not an issue, since that can be configured; in some cases easily, in some cases less so, but it can be done, and then it’s only a question of how easy it is to add data sources. 

The bigger issue is the regular management and maintenance tasks, which can be a pain if you have to log into and out of multiple environments to check service status or perform updates. If your system management is fragmented, it’s hard to acquire an overall picture of the whole system, and that has implications for troubleshooting and optimization. 

Companies using orchestration systems like Kubernetes and Terraform to centralize and automate management tasks find that it enhances their control over resource consumption. With an orchestration system, companies can utilize a mixed set of services as easily as just one. These systems make it possible to centrally manage both individual services and the way the services interact. This gives a better overview of system status and performance and makes it quicker and easier to troubleshoot and fix issues. And of course, access management can also be centralized to an orchestration system, removing yet more regular upkeep tasks.

Kubernetes and Terraform also provide greater horizontal scalability. Instead of having to build endpoints into each service individually, it’s enough to build them in the orchestration system–and both of them also come with a wide selection of ready-made open source connectors, saving companies the trouble of writing their own.
This is how the benefit from orchestration combines with the benefit derived from increased agility in the marketplace and takes both to a whole new level. At Aiven, we see this all the time. Large numbers of our customers use Kubernetes and/or Terraform to save time, money and effort in terms of both system maintenance and product development.

Wrapping up

The common denominator for companies adopting managed cloud databases and combining it with an orchestration system is that they’re able to reduce their expenses markedly, due to a combination of reduced capital expenditures and smaller operating costs, even when taking into account the fees paid for their database management tasks. 

With a DBaaS in use, companies can also stay agile and responsive, and continuously evaluate and adjust their business model as the circumstances change. With OVHcloud’s managed cloud databases, powered by Aiven, you don’t have to worry about your database infra or administration, including set up, backups, scalability and security. This saves you time and money, so you can focus on building and deploying the cloud-based applications you want.

One way to tackle that is to define and follow good life cycle policies.

What is a life cycle policy

A life cycle policy is a set of guidelines, written documents, describing the different steps in a product’s life, including its end of life.

Why you should have a life cycle policy

When you read the news, it is not uncommon to hear from here and there that company XYZ got hacked through an old software, outdated for ages.

Having a good life cycle policy guarantees you never have to deal with outdated software. It defines when you must upgrade your software and when you must stop using an old version and migrate to a new one.

Benefits are multiple:

• You don’t have to rely on unmaintained software
• You reduce attack perimeter, as newer versions have security patches applied
• You benefit from new versions’ features, allowing you to accelerate innovation

Of course there are downsides. The biggest one is having to stick with it. A life cycle policy is a written document and as such has value only if people read it and apply its recommendations.

Why you should automate it

A wise man once told “A good sysadmin is a sysadmin doing nothing”. Not because he is lazy, nor because he is bad at his job and got his credentials revoked, but because he automates all painful tasks, to focus on tasks with added value.

It also removes one major cause of errors: the human factor. An automated job will never mistype a command line or click on the `destroy` button, instead of the `upgrade` one.

Last but not least, it is a good way to make sure your carefully crafted document is not redirected to `/dev/null` in the following months.

Bonus: you also free a lot of time to automate new things !

How we did it

Our policy on cloud databases (available here: https://docs.ovh.com/gb/en/clouddb/managed-db-life-cycle-policy/) forces us to warn our customers at least 30 days before a major upgrade by e-mail. This email contains the day the upgrade will take place on.
We also have an internal mechanism allowing our customers to upgrade by themselves, using our API.
Combining these two bricks were all we had to do, with minor adjustments:

• When sending a email, we keep track of the planned date of the operation.
• An automated job is triggered every day, checking what operations are due this day and launching them.

Almost everyday, our job realises there is no major upgrade planned and goes back to sleep. But if a single operation is found, this job calls our API, triggering this upgrade. No human needed.

This process is very simple, yet very effective. We already had all the building blocks ready, from sending an email to API calls for major upgrades. All we had to do was to link all these operations together in a nice workflow. This allow us a very fine tuning of how many operation we want being done everyday, down to the instance.

Want to try the process for the first time ? Easy, trigger it on one of our test instance.
Is this process reliable and scalable ? Enough to upgrade a batch of 10 000 instances in 30 days with no human intervention !

One last word on an automated life-cycle policy

At Web Cloud Databases, with thousands of instances, 4 different DBMS and a total of 12 major versions, automation is the key.

In May 2021, for this process’ firsts steps in this world, we upgraded 6K instances in a week. The only human interaction needed was to launch a script to inform all affected customers.

Figures of this use-case ? 6000+ instances updated, 0 website down, 1 human needed for 5 minutes.

A large choice of “vanilla flavor” DBMS

The market is brimming with database engines, we will offer you this large choice over the coming months. Our Public Cloud Databases are of course fully managed and compatible with the full OVHcloud portfolio including bare metal for instance.

Did you know that delegating the setup and daily operations of your database cluster could help your improve your performance by 15% to 20%? That’s a pretty good argument to move your database to the cloud if it wasn’t already the case!

You might still wonder how these DBaaS differ from existing market offerings?

• Aside from MongoDB, all other engines are open-source so that you’ve got no risk of vendor lock-in and can adapt your technology strategy in the long run. We partner with the leader of open-source DBaaS, Aiven, to deliver this full portfolio.
• What’s more, we deliver the official, full and updated experience or “vanilla flavor” of each DBMS, and not a fork in our cloud environment. It means that you benefit from all the features, updates and community innovation.
• They are hosted in our trusted Cloud-Act free cloud infrastructure – in Europe and Canada today, and soon in other parts of the world – to ensure you full data privacy and infrastructure compliance. 
• Last, but not least, after the free beta, their pricing is fully transparent, predictable and much cheaper than competition. Why don’t you check our price list by yourself? What you see is what you pay! All costs including outgoing traffic are included. Don’t forget to multiply the price by the number of nodes!

You voted, we delivered

We know you have been waiting for these managed databases in the public cloud for some time, so we asked you which were your favorite ones and delivered them first. After 3 weeks of usage, here is our beta testers’ podium:

1. Almost half of our first beta testers use PostgreSQL. It’s the most advanced, extensible and highly secure relational database that you use for mission critical business applications. 
2. Another big third of our users are on MySQL, one of the most popular transactional engines for eCommerce platforms, but also an ideal choice for many web applications. 
3. Redis is the third favorite database with almost 15% adoption. It is frequently used for website caching and personalization, chat and messaging apps but also in the gaming industry for leaderboards.
4. The last few percent of users are on Apache Kafka , the must have distributed data streaming platform. Its horizontal scalability is useful  for IoT data ingestion, real-time analytics as well for event-driven applications.

But what do Kafka, MySQL, PostgreSQL, OpenSearch and Redis do?

Long story short: too many uses cases for one blog post! But let’s dive a bit into the main concepts and key differentiators.

Imagine that you are literally an e-commerce website… At least try to imagine 😀  It may be a little silly, but a good way to understand.

It’s almost Christmas, plenty of customers are creating accounts, ordering gifts, putting their credit cards. As a website, you cannot lose their information, you want to store it precisely. Here come relational databases to the rescue!

Relational databases with MySQL and PostgreSQL

Relational databases are widely adopted worldwide, with pretty simple tables, with columns and rows. Each table can be linked to others, with relationships, it’s called a schema.

The most famous ones are MySQL, PostgreSQL, MariaDB and SQL server.

The idea is always the same, tables and relations, with a goal to guarantee data validity despite errors, power failures.

Imagine that you, the website, receive an order for a new gift; you ask for a payment of 50 euros. The customer is paying, but just for 2 seconds your database is flapping and unreachable.

As a website, do you want approximate data? Do you want to be almost certain that the customer has payed? Certainly not! You want to be 100% confident about the transaction in your databases.

Relational Databases provide ACID guarantees (for Atomicity, Consistency, Isolation and Durability) that make them the best solution to deal with this kind of use case.

MySQL and MariaDB are often used for simple website and web application, such as WordPress.

PostgreSQL is more production grade-oriented, with full ACID compliance, advanced security and high availability (HA mechanism, and various extensions for various use case (geographical data, metrics, …).

Also, PostgreSQL is the only database system in the list owned by no company. It’s owned by the PostgreSQL community, you and me. Unlike MySQL that’s the property of Oracle for example.

Let’s come back to our e-commerce scenario!

It’s Christmas time for our website, your are now able to receive and store correctly your products, the orders, the bills… neat!

But wait… Now more and more customers are coming to you, asking ALWAYS for the same price and description of this wonderful pair of socks. Worst, they put a few products in the basket, but don’t buy them.

You are slowed down by storing and giving information from your local storage (images). In the same way, pushing in database information that you would not need to keep (basket) is frustrating. Last but not least, tons of requests in your relational databases (“SELECT Articles in discounted_prices…”) are always the same, giving the same results…

Grrr! It’s really painful to store this kind of ephemeral information. What if you could store and retrieve this data somewhere very, very fast?

Hello Redis™*! your best friend for caching data

The most handy database engine to answer your need is Redis™*.

Redis has two main characteristics: first, it stores data in volatile memory (RAM), not in classic storage such ad SSD drives. Second, it’s not a relational database, but a key-value store. You can store various type of data, such as HTML code, images, user sessions, gaming leaderboards, chat sessions, …

Each time you have to keep in mind that this data “can” be lost and you will store/retrieve it with huge performance.

Most popular use cases are website performance improvement, databases caching, gaming leaderboard, chat tools,.. Twitter, Snapshat, instagram, AirBnB, Pinterest.. they all use Redis!

The most handy database engine to answer your need is Redis.

Redis has two main characteristics: first, it stores data in volatile memory (RAM), not in classic storage such ad SSD drives. Second, it’s not a relational database, but a key-value store. You can store various type of data, such as HTML code, images, user sessions, gaming leaderboards, chat sessions, …

Each time you have to keep in mind that this data “can” be lost and you will store/retrieve it with huge performance.

Most popular use cases are website performance improvement, databases caching, gaming leaderboard, chat tools,.. Twitter, Snapshat, instagram, AirBnB, Pinterest.. they all use Redis!

Kafka, to avoid kafkaian situations

Apache Kafka, is the world’s most famous streaming tool. Imagine a highly-available and highly efficient platform, allowing you to connect to dozen data sources, “snif” the data in real time with near zero latency and push it somewhere else. That’s Kafka which unlike Redis is used a bit everywhere.

You don’t store information in Kafka, you connect Kafka to information sources. Then, of course, this data can be reused.

Main use cases are messaging between applications (it acts as a buffer), real time analytics and real time data pipelines, aggregation of metrics and logs (imagine a company with 1.5 million website ands 250’000 servers to monitor, for example ).

OpenSearch to keep en eye on everything

OpenSearch is an open source fork of Elasticsearch, allowing you to ingest data, such as all the logs of your website and easily analyze them.

Perfect for us to follow all the 404 errors for example! Wonderful dashboard in minutes, in real time!

Here is an example made with OpenSearch sample data:

The A-team of data ecosystem

As a website, you can think of relational databases, Redis, Opensearch and Kafka as the A-team. They are not similar at all; each of them has a specific role to play in the adventure.

At the end, I love it when a plan comes together!

Learn more about these use cases with Aiven’s Cofounder and CTO, Heikki Nousiainen, at Ecosystem Experience!

*Redis is a trademark of Redis Ltd. Any rights therein are reserved to Redis Ltd. Any use by OVH SAS is for referential purposes only and does not indicate any sponsorship, endorsement or affiliation between Redis and OVH SAS.

After hundreds of cluster delivered for free, dozens of pull requests, tons of constructive discussions with customers and partners,… we are live!

We are really pleased to announce our Public Cloud Databases for MongoDB offers on General Availability 😀

And today is a good day! MySQL, PostgreSQL, Kafka, … are now also available for FREE in Beta!
We will talk about it in another blog post, that’s another story 😀

About MongoDB offers, what does it mean? How does it work? Why you should use it? Let’s find out together!

What do we solve?

Installing MongoDB is an easy operation if you have basic computer administration knowledge. But that’s the easy part.

Operating databases on a daily basis is way more challenging. Databases are quite often critical for your use case, and taking care of them ends up being a full time job. Instead of deploying and maintaining your databases yourself, what if a dedicated team took care of the setup and initial state-of-the-art configuration, securization, and maintenance in operational conditions?

That’s quite appealing. It’s not new in the “Cloud world”, many competitors built Databases as a Services solutions, I guess you’ve heard about the most famous ones, aka Amazon RDS or Google Cloud SQL.

But this is new for OVHcloud in our Public Cloud ecosystem, and a very important step!

We are now able to solve a major customer challenge: fast deployment of database clusters, with scalability, security and resiliency.

Let’s deep dive into the detailed features!

What’s in the box for MongoDB offers?

Official MongoDB software and license

MongoDB is recognized worldwide as the best document-based database engine. This software is available in two versions:

• MongoDB Community: the most common MongoDB version, under Server Side Public Licence (SSPL). To put it simply, it’s free unless you use it as a “Program as a Service”;
• MongoDB Enterprise: same core but advanced features for security, backups, auditing tools, BI connectors, MongoDB consulting… Perfect for advanced use-cases.

At OVHcloud, we made a strong partnership with MongoDB, allowing us to provide both Community and Enterprise licenses “as a Service”. That’s indeed a very good news for customers since we are able to provide “vanilla”, genuine version of MongoDB.

It ensures that you get the full MongoDB experience, and are able to move in / out OVHcloud if required. It’s not a fork, like AWS DocumentDB (see the MongoDB compatibility test) or Azure CosmosDB. Hence, you benefit from the latest MongoDB features as well as the official MongoDB documentation, and drivers work natively.

As I am writing, offers with Community License are in General Availability, and Enterprise will come later.

Easy to start, easy to use, easy to scale

Let’s see how simple it is!

Through our control panel, you can launch a MongoDB Database instance in one minute. Select a region, a version, a service plan, and voila! After a few minutes, your cluster is up and ready, allowing you to manage security rules, users, backups, logs and metrics.

It’s also easy to scale; you can upgrade to a higher service plan, add up to 8 nodes per cluster, and pay what you use- true for every public cloud product -on a hourly basis.

Of course, you can  delete your services in a few clicks, allowing you to try the products for few bucks.

Managed by OVHcloud

Again, our aim is to simplify your life and cloud adoption. What we do for you:

• Install MongoDB clusters, based on your choices (versions, nodes, and co);
• Operate the subjacent Linux system (patches, hotfixe, upgrades, …);
• Operate the DBMS software: provide minor and major DBMS versions;
• Operate the subjacent hardware;

What you remain responsible for:

• your data,
• your structures,
• your code
• performance monitoring…

In short, what’s “in” the database instance. We cannot replace you on this 😀

High-Availability and automatic backups

We built 3 service plans: Essential, Business and Enterprise.

All the plans come with automated daily backups, encrypted and stored ouside the datacenter of the cluster.

Starting with Business plans, High-Availabilty is provided natively with 3 nodes by default (Replica Set), maximum 8 nodes. Each node includes a copy of your data to keep it safe. In case of a node failure, your cluster is still up and ready.

Currently we do not provide clusters replicated across multiple regions but we are working on it.

Logs and Metrics Dashboards

To simplify your obvervability tasks, we provide live logs and main metrics in our control panel.

Our mid-term goal is to expose metrics and log to third-party tools.

Last but not least: simple billing! Cloud without headaches

We really focused on simplifiying your cloud adoption, and one of the biggest drawbacks for customers is to predict and control the cost of their cloud infrastructure.

Take a look at our pricing page, we’ve done our best to keep it simple: no hidden cost, everything is predictive and included:

• Databases services with included storage;
• Guaranteed resource performance over time (no burst, no IOPS to buy)
• Data transfer In/Out
• Daily Backups
• logs and metrics

Zero risk of overspending!

Plenty of use-cases

MongoDB is a document-based database, meaning that you can store heterogeneous data in it, not in a classic SQL way (tables, columns, lines) but in a “flat” way like a JSON file.

Many use-cases benefits from this data storage and queries, such as IoT, Real-time data analytics, Single View and much more. The official MongoDB website details the most important ones.

To get a managed MongoDB, what are the other options in cloud providers?

Let’s have a quick look at competitors. If you are looking for a managed MongoDB cluster under European legislation only, OVHcloud and CleverCloud are the only providers offering it.

Choosing a provider will require more than one table, but here is a quick overview. If you are not aware, MongoDB also provides managed services, it’s called MongoDB Atlas. really impressive offers from a feature perspective. You can host them in AWS, Google Cloud or Hosting. Since it’s not a cloud provider offer, I decided to not compare this last one.

The end! Useful links

I really hope that offers and features are clearer now, and like us that you are over hyper about the data future a OVHclooud. I you need more, here

I really hope that offers and features are clearer now, and that you are as enthusiastic as we are about the data future at OVHcloud!

Should you want to learn more, here we are:

French website

English Website

Official documentation

Discuss with us on our Discord servwr!

We’ve just started providing managed databases as a service in our Public Cloud offer, starting with the wonderful MongoDB engine.

Many others will follow in the coming months. This means so much to us and also for you, our fellow customers. Let me tell you what’s going on.

OVHcloud and databases, a long love-story

If you ask any company around the world about their most critical services in their architecture, there is no doubt that databases will be at the epicenter, almost as critical as the coffee machine 😀. Your data is usually at the heart of your use cases, your websites and applications, your company business.

What’s not so obvious is pushing your data in safe databases. Installing MySQL, for example, is one thing. However, doing it state-of-the-art with strong security, resiliency, and performance is another. It takes knowledge and patience.

That’s why today every cloud provider is building managed databases services with one goal: to free customers from the hard stuff – setup, security, high-availability, monitoring, backups, updates, …

Since we host more than 3 million websites with more than 1.5 million databases behind them, we have solid knowledge and experience of databases (curious ? more details here !). Same for our internal databases clusters, it’s a love story. But up until now, nothing had been provided to our customers in our Public Cloud.

Databases: a missing brick in our Public Cloud ecosystem, now resolved!

If you’re not familiar with the public cloud, I’ll sum it up quickly: you start a new project, add a payment method, then you’re able to launch multiple and various services (such as virtual machines, storage, big data and machine learning platforms) … and at the end of the month, you only pay for what you consumed (per hour or per minute).

It’s called “Public” because some parts, like network equipment or physical servers, is shared with other customers (unlike a Private Cloud). It’s secure. No one has access except for you. But isolation is mostly made with software.

So, back on topic, we launched our OVHcloud Public Cloud over 6 years ago, and have improved our inner ecosystem brick by brick, month by month, year by year. We started with virtual machines, block storage, object storage, cold storage, then added containers (Kubernetes clusters, private registries, ), then network (load balancers, ..) then big data and AI/machine learning, but one major brick was missing: managed databases.

To be fully transparent, we were thinking about it for a very long time, around 4 years if I’m correct. We felt like Indiana Jones trying to avoid traps during the Holy Grail quest. A long and treacherous path. But why so long ?

Well, data volumetry is growing faster than ever, usages are multiplying exponentially, same for the solutions to store and use them. About 10-15 years ago it was about classic domination, like Oracle, MS SQL server, PostgreSQL and MySQL and voila. Four years ago things were way more divided and complex, as Redis, MongoDb, Cassandra, ElasticSearch were already in high demand.

We feel kind of like spectators of a moving landscape, it’s moving so fast that it’s hard to catch up. Don’t blink or you’ll miss something! 😉

Four years ago, we wanted to provide, not only 1 or 2 databases engines, but the most appreciated by our customers, the most requested and loved by communities, and it’s quite complex. Each of them has their own method to security, to perform backups, to setup high resiliency, to manage users and roles, to perform updates, requirement of various hardware, … a brain teaser when you want to industrialise it!

To catch up with the market, we needed to find the best companies outside of OVHcloud to help us on this quest.

We found MongoDB Inc, providing not only a databases engine, but also high quality tools and support to help customers (and cloud providers like OVHcloud) to deploy, maintain and monitor large MongoDB infrastructures. From day one, it worked fast and (furiously) resilient by design.

Introducing MongoDB offers this week, and right now it’s free!

Yeah! Yay! Woohoo! We are thrilled to announce managed databases services in our public cloud, now with MongoDB! That’s really good news.

In just a few clicks (or 1 API call) you’ll have a managed MongoDB cluster with a Community or Enterprise licence, delivered in under 5 minutes. We are one of the few cloud providers worldwide to deliver official MongoDB software, not a fork, as a service.

We even created three service plans to aide you on your journey: Essential, Business and Critical. Full details on the website!

And you know what? It’s free during the Beta phase (100% free, excluding official support or SLA, not qualified for production). Just connect to our OVHcloud control panel and launch Essential or Business MongoDB clusters. Enterprises ones will be available shortly after, in July.

As an OVHcloud customer, you’ll be able to focus on your core business, and rely on us for all your database infrastructure and maintenance needs.

We will take care of the rest like setup, installation, security and then maintain these services operationally.

Don’t get me wrong: you’re still responsible for your own data and your queries. It’s not magic. But we’ll take care of the infrastructure part, which is oh so nice. We added extra features such as daily backup (archived in another geographical region), high availability (MongoDB replica set), and public or private network (vRack). We’ll have more features soon too! For example, observability tools are currently underway and in development.

But what about other database engines?

MySql, PostgreSQL, Redis, Cassandra, M3db, Kafka, ElasticSearch,  … How appealing does this list look!? 😀

Even more good news! We’ve also teamed up with a pure player in the databases field as a technological partner for Public Cloud Databases and streaming tools. This means that MongoDB is not only a serious first step, but in a few short months we will add even more engines!

Their core-business is doing databases and they handle it like no one else. We will be able to catch up and provide to you, our customers, a much bigger and better Public Cloud portfolio. They will simplify industrialization and all the engine-related specifics.

Do not worry! We will come back here at the end of June to share our tips and tricks and lovely insight.

We really hope that you’ll enjoy this new adventure with OVHcloud. Try for yourself and please share your feedback on OVHcloud Discord #database-MongoDB!

In this post, I explained how we had to push proxySQL past its limits.

Julien explained here how difficult it can be to hijack pieces of a software program, and make it work for a very specific use case. One so specific that we just could not push it upstream, as we were breaking some assumptions on the base code (something like, “look, we don’t use this, maybe we can divert it, to make it match our use”).

But sometimes, as we dig deeper and deeper through lines of code, pushing the limits of open-source software programs further and further, we reach bugs.

Here are two real-life case where we had an unexpected behaviour from ProxySQL, resulting in a patch for MariaDB/MySQL.

1. The importance of quoting

An unexpected behaviour

When we first started using ProxySQL, we had an unexpected behaviour. ProxySQL used to hang. No warning, no precursor. A hang means that ProxySQL cannot process your requests, causing your websites to be unavailable as a result.

Digging

Sysadmins know that when in doubt, check the logs. That’s what we did, and we noticed this:

[ERROR] Detected a broken connection during SET NAMES on 192.168.59.272 , 3306 : 2019, Can't initialize character set (null) (path: compiled_in)

So we took a closer look at the logs, tried to replicate the behaviour, and finally came across this:

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'binary' at line 1

This is clearly an error returned from the MySQL server. As this log line was generated on the ProxySQL, it means the error was between our ProxySQL and the MySQL server.

Finding gold

Julien worked hard — he read tonnes of logs, traced many PIDs, and tracked the problem down to THE triggering case — issuing this command:

set names binary COLLATE binary;

A collation is a kind of rule-set for comparing strings. It can define, for example when sorting alphabetically, if A comes before a, if é should be treated as e or not, or where œ should be in the alphabet.

You can read more about it on MariaDB’s Knowledge Base.

The fix

After submitting an issue on ProxySQL’s bug-tracker with what we discovered, and proposing a patch, the author of ProxySQL had a look and confirmed the bug.

While writing ProxySQL, René Cannaò did what all developers do — he followed MariaDB’s connector documentation. And the bug was from here:

According to the documentation on SET NAMES syntax (https://dev.mysql.com/doc/refman/5.7/en/set-names.html) :

charset_name and collation_name may be quoted or unquoted

This doesn't seem true for "SET NAMES binary COLLATE binary" , that requires the collation name to be quoted.

(https://bugs.mysql.com/bug.php?172id=93692)

The butterfly effect

So, from a hang on our infrastructure, we reported a bug to the project’s creator, who then traced it to a bug in a MariaDB connector, followed it up to its parent MySQL, and fixed it upstream. The bug was closed in early 2020.

In the meantime, we worked with Mr Cannaò to provide a workaround (basically forcing the collation name to be quoted in ProxySQL’s code).

2. When wires get crossed

While writing this article, I remembered another funny bug. I decided to check this bug status, and noticed we never reported it. My mistake is now fixed.

An unexpected behaviour

When using some real but exotic charsets on ProxySQL (we also love tricky scenarios, so we try to test them as much as possible), we had a ProxySQL error, stating we were using an incorrect charset.

MySQL_Session.cpp:2964:handler(): [WARNING] Error during query on (42,192.168.59.272,3306): 1267, Illegal mix of collations (utf8_general_ci,IMPLICIT) and (dec8_swedish_ci,COERCIBLE) for operation '='

Digging

Of course, first of all we checked that we were using a real and legit charset and associated collation. We double- and triple-checked that we should be allowed to use `dec8_swedish_ci`.

We decided to have a look at the connector’s source code.

Finding gold

If you are curious about it, you can have a look at older versions of the `libmariadb/my_charset.c` code, starting from line 541. You will notice that `dec8_swedish_ci` is nowhere to be found. But if you look closely, you will notice dec8_swedisch_ci. Dear friends from Sweden, you were not the only ones with a typo.

The fix

When fixing our issue with exotic charsets, we applied a custom patch on our custom build of ProxySQL. In the heat of the moment, we delayed the bug reporting.

We forked the mariadb-corporation / mariadb-connector-c GitHub repository, fixed some typos, and proposed a pull request — which was then merged mid-February.

The butterfly effect

Everyone is prone to typos, and making mistakes. We encountered some, they were fixed and ported for everyone.

Conclusion

As a leading web hosting provider in Europe, we face the Law of truly large numbers (https://en.wikipedia.org/wiki/Law_of_truly_large_numbers). TL;DR: Each and every event with a non null occurring probability will happen.

We host more than 1.3 million websites. Let’s assume that 0.0001% of our customers can trigger one single specific bug — we have at least 1 customer who will trigger it. The question is not if, but when we will have to deal with it, and how to do so.

Wondering how we noticed that we had hit the Law of truly large numbers? Stay tuned, because we will write about this soon.

Sharing is caring

OVHCloud loves open-source, and loves sharing. This is not the first time we contributed to the open-source world, and definitely not the last.