We can compare it to a game with different rule sets, on which every player must agree before playing.

The UNO analogy

Let’s make an analogy with the UNO card game. There is many variants, but we can sum them up like this:

Before starting to play, one defines uno_mode=DOUBLE,INTERCEPTION, which means the STRAIGHT_FLUSH rule does not apply. If this rule is activated later in the game, it may cause problems around the table, some players may have been disadvantaged in the previous rounds and may not want to play any more.

A card game to manage data?

SQL mode works like this, but instead of specifying how to play cards, it specifies what to do in certain situations:

• Is “2020-11-00” a valid date (NO_ZERO_IN_DATE)? It changes data validation performed by the DBMS.
• Is SELECT name FROM users GROUP BY first_name a valid query (ONLY_FULL_GROUP_BY)? It changes the allowed syntax.

When MariaDB 10.2 and MySQL 5.7 were released, the rules have changed:

We can note that newer versions kept old rules, but new ones were added. That’s why when upgrading from MariaDB 10.1 to MariaDB 10.2 for instance, one must specify to its database to play with old rules, and not to enforce STRICT_TRANS_TABLES, ERROR_FOR_DIVISION_BY_ZERO.

Concretely

Actually, if my website should store the result of a division, but the denominator is `0`, my database with the MariaDB 10.1 default SQL mode database will allow it and store the result as NULL. My database with the MariaDB 10.2 default SQL mode will also allow and store a NULL value it but will raise a warning. If strict mode is also enabled, MariaDB 10.2 will throw an error and store nothing. Thus, changing SQL mode can break websites and should be done carefully.

As we cannot check every line of code of every website of each of our customers, when an DBMS major upgrade changing the default SQL mode is planned, OVHcloud activates the legacy SQL mode ensuring your website will not be affected. Of course, you can switch to the new default SQL Mode in the “Configuration” tab of your CloudDB. But keep in mind this is for experienced users and OVHcloud advises to stick with the default sql_mode, unless your database was upgraded from a previous version with a different default sql_mode.

Official Documentation:
MariaDB: https://mariadb.com/kb/en/sql-mode/
MySQL: https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html

One way to tackle that is to define and follow good life cycle policies.

What is a life cycle policy

A life cycle policy is a set of guidelines, written documents, describing the different steps in a product’s life, including its end of life.

Why you should have a life cycle policy

When you read the news, it is not uncommon to hear from here and there that company XYZ got hacked through an old software, outdated for ages.

Having a good life cycle policy guarantees you never have to deal with outdated software. It defines when you must upgrade your software and when you must stop using an old version and migrate to a new one.

Benefits are multiple:

• You don’t have to rely on unmaintained software
• You reduce attack perimeter, as newer versions have security patches applied
• You benefit from new versions’ features, allowing you to accelerate innovation

Of course there are downsides. The biggest one is having to stick with it. A life cycle policy is a written document and as such has value only if people read it and apply its recommendations.

Why you should automate it

A wise man once told “A good sysadmin is a sysadmin doing nothing”. Not because he is lazy, nor because he is bad at his job and got his credentials revoked, but because he automates all painful tasks, to focus on tasks with added value.

It also removes one major cause of errors: the human factor. An automated job will never mistype a command line or click on the `destroy` button, instead of the `upgrade` one.

Last but not least, it is a good way to make sure your carefully crafted document is not redirected to `/dev/null` in the following months.

Bonus: you also free a lot of time to automate new things !

How we did it

Our policy on cloud databases (available here: https://docs.ovh.com/gb/en/clouddb/managed-db-life-cycle-policy/) forces us to warn our customers at least 30 days before a major upgrade by e-mail. This email contains the day the upgrade will take place on.
We also have an internal mechanism allowing our customers to upgrade by themselves, using our API.
Combining these two bricks were all we had to do, with minor adjustments:

• When sending a email, we keep track of the planned date of the operation.
• An automated job is triggered every day, checking what operations are due this day and launching them.

Almost everyday, our job realises there is no major upgrade planned and goes back to sleep. But if a single operation is found, this job calls our API, triggering this upgrade. No human needed.

This process is very simple, yet very effective. We already had all the building blocks ready, from sending an email to API calls for major upgrades. All we had to do was to link all these operations together in a nice workflow. This allow us a very fine tuning of how many operation we want being done everyday, down to the instance.

Want to try the process for the first time ? Easy, trigger it on one of our test instance.
Is this process reliable and scalable ? Enough to upgrade a batch of 10 000 instances in 30 days with no human intervention !

One last word on an automated life-cycle policy

At Web Cloud Databases, with thousands of instances, 4 different DBMS and a total of 12 major versions, automation is the key.

In May 2021, for this process’ firsts steps in this world, we upgraded 6K instances in a week. The only human interaction needed was to launch a script to inform all affected customers.

Figures of this use-case ? 6000+ instances updated, 0 website down, 1 human needed for 5 minutes.

In this post, I explained how we had to push proxySQL past its limits.

Julien explained here how difficult it can be to hijack pieces of a software program, and make it work for a very specific use case. One so specific that we just could not push it upstream, as we were breaking some assumptions on the base code (something like, “look, we don’t use this, maybe we can divert it, to make it match our use”).

But sometimes, as we dig deeper and deeper through lines of code, pushing the limits of open-source software programs further and further, we reach bugs.

Here are two real-life case where we had an unexpected behaviour from ProxySQL, resulting in a patch for MariaDB/MySQL.

1. The importance of quoting

An unexpected behaviour

When we first started using ProxySQL, we had an unexpected behaviour. ProxySQL used to hang. No warning, no precursor. A hang means that ProxySQL cannot process your requests, causing your websites to be unavailable as a result.

Digging

Sysadmins know that when in doubt, check the logs. That’s what we did, and we noticed this:

[ERROR] Detected a broken connection during SET NAMES on 192.168.59.272 , 3306 : 2019, Can't initialize character set (null) (path: compiled_in)

So we took a closer look at the logs, tried to replicate the behaviour, and finally came across this:

ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'binary' at line 1

This is clearly an error returned from the MySQL server. As this log line was generated on the ProxySQL, it means the error was between our ProxySQL and the MySQL server.

Finding gold

Julien worked hard — he read tonnes of logs, traced many PIDs, and tracked the problem down to THE triggering case — issuing this command:

set names binary COLLATE binary;

A collation is a kind of rule-set for comparing strings. It can define, for example when sorting alphabetically, if A comes before a, if é should be treated as e or not, or where œ should be in the alphabet.

You can read more about it on MariaDB’s Knowledge Base.

The fix

After submitting an issue on ProxySQL’s bug-tracker with what we discovered, and proposing a patch, the author of ProxySQL had a look and confirmed the bug.

While writing ProxySQL, René Cannaò did what all developers do — he followed MariaDB’s connector documentation. And the bug was from here:

According to the documentation on SET NAMES syntax (https://dev.mysql.com/doc/refman/5.7/en/set-names.html) :

charset_name and collation_name may be quoted or unquoted

This doesn't seem true for "SET NAMES binary COLLATE binary" , that requires the collation name to be quoted.

(https://bugs.mysql.com/bug.php?172id=93692)

The butterfly effect

So, from a hang on our infrastructure, we reported a bug to the project’s creator, who then traced it to a bug in a MariaDB connector, followed it up to its parent MySQL, and fixed it upstream. The bug was closed in early 2020.

In the meantime, we worked with Mr Cannaò to provide a workaround (basically forcing the collation name to be quoted in ProxySQL’s code).

2. When wires get crossed

While writing this article, I remembered another funny bug. I decided to check this bug status, and noticed we never reported it. My mistake is now fixed.

An unexpected behaviour

When using some real but exotic charsets on ProxySQL (we also love tricky scenarios, so we try to test them as much as possible), we had a ProxySQL error, stating we were using an incorrect charset.

MySQL_Session.cpp:2964:handler(): [WARNING] Error during query on (42,192.168.59.272,3306): 1267, Illegal mix of collations (utf8_general_ci,IMPLICIT) and (dec8_swedish_ci,COERCIBLE) for operation '='

Digging

Of course, first of all we checked that we were using a real and legit charset and associated collation. We double- and triple-checked that we should be allowed to use `dec8_swedish_ci`.

We decided to have a look at the connector’s source code.

Finding gold

If you are curious about it, you can have a look at older versions of the `libmariadb/my_charset.c` code, starting from line 541. You will notice that `dec8_swedish_ci` is nowhere to be found. But if you look closely, you will notice dec8_swedisch_ci. Dear friends from Sweden, you were not the only ones with a typo.

The fix

When fixing our issue with exotic charsets, we applied a custom patch on our custom build of ProxySQL. In the heat of the moment, we delayed the bug reporting.

We forked the mariadb-corporation / mariadb-connector-c GitHub repository, fixed some typos, and proposed a pull request — which was then merged mid-February.

The butterfly effect

Everyone is prone to typos, and making mistakes. We encountered some, they were fixed and ported for everyone.

Conclusion

As a leading web hosting provider in Europe, we face the Law of truly large numbers (https://en.wikipedia.org/wiki/Law_of_truly_large_numbers). TL;DR: Each and every event with a non null occurring probability will happen.

We host more than 1.3 million websites. Let’s assume that 0.0001% of our customers can trigger one single specific bug — we have at least 1 customer who will trigger it. The question is not if, but when we will have to deal with it, and how to do so.

Wondering how we noticed that we had hit the Law of truly large numbers? Stay tuned, because we will write about this soon.

Sharing is caring

OVHCloud loves open-source, and loves sharing. This is not the first time we contributed to the open-source world, and definitely not the last.