Feedback on a one-week correction operation carried out on our entire KVM fleet, following a patching strategy, with the central objective of minimising the impact on our customers’ services.

A vulnerability in the virtualisation subsystem

On Tuesday 7 July, early in the afternoon, a security alert was reported regarding CVE-2026-53359: a use-after-free vulnerability affecting the shadow-paging subsystem of KVM x86 in the Linux kernel. The vulnerability, which was several years old, was made public on 6 July, with blogs from other cloud providers starting to appear on 7 July.

KVM is the virtualisation engine on which the vast majority of the instances hosted at OVHcloud are based. The mechanism is as follows: when an external modification of a Page Directory Entry (PDE) occurs, the RMAP entry can retain a reference to a memory page that has already been freed. The kernel then de-references this obsolete page, which can lead to a hypervisor crash or, worst-case scenario, a host-side privilege escalation. The exploit is reproducible: an internal test on an unpatched host causes a crash in about two minutes.

All Linux x86 kernels prior to the commit of the patch are affected, regardless of distribution. The official patch is backported to our production Debian kernels.

There are three risks to consider:

• A VPS customer uses the exploit to crash the host running their virtual machine, causing a crash and the uncontrolled reboot of several hundred customer virtual machines.
• Identical scenario on Public Cloud instances: the impact is similar, but the number of virtual machines per host is more limited, and the virtual machines are more powerful and used for more sensitive systems (databases, queue manager, load balancer, etc.). These systems are often used in composite application architectures with complex dependencies.
• Likelihood of a future publication of a host takeover exploit, which would immediately raise the risk level unacceptably high, due to the impact on customer data confidentiality and infrastructure integrity.

For OVHcloud, the scope covers tens of thousands of hypervisor host servers, housing about one million virtual machines. The question is therefore not whether to patch, but how to carry out this operation on such a large scale, knowing that zero customer impact is not achievable.

Options for mitigating risk

1. Wait for official patched kernels

This option would leave us dependent on a third-party timeframe, leaving us in an at-risk status for an uncontrolled period of time. We therefore quickly ruled this option out.

2. Live patch

Applying a live patch requires authorising this type of operation in the kernel configuration, which allows (by design) ad-hoc modification of its behaviour, but results in a reduction in its hardening level, limiting its compromise detection capabilities. Furthermore, the live patch is a sensitive procedure, which could lead to an unstable fleet status. It is an option that buys time while waiting for a long-term solution. We decided not to make this compromise, as it would have sharply increased the risk if an exploit for taking control of the host was published.

3. Mitigation by disabling nested virtualisation

Disabling nested virtualisation on the hosts would make the exploit inoperative. We do not have visibility into how our customers use this feature, so it is not possible to determine the impact on customer services, and this feature is necessary to maintain the ability to live migrate an instance from one physical host to another. As a result, we quickly ruled this out.

4. Live migrate

This involves organising a live migration of virtual machines, from vulnerable hosts to empty hosts that have been patched in the meantime. This option is very satisfactory in terms of service continuity because the migration is done without any impact on the virtual machines, with the exception of reduced performance during the migration. This option takes a long time due to the host-to-host virtual machine copy, meaning that it is not realistically usable due to the scale of the fleet and the desire to protect our customers within a few days rather than several months. We decided to keep this option for certain critical virtual machines only, as each live migration causes a significant delay in the execution of batches.

5. Backport the patch on our kernels and reboot all hosts

We ultimately chose this option, which will be detailed further in this article.

Tuesday afternoon: activation and organisation of the crisis unit

As soon as the vulnerability was confirmed, the priority was to build a structured and coordinated response; the analysts responsible for this initial analysis quickly understood the challenges of the coming days. The information was distributed internally at the beginning of the afternoon. Several coordination spaces were created: one for technical coordination, one for crisis coordination, one for US operations, one for customer communication and support. At the same time, the Kernel teams prepared and backported the patch, with the first patched kernel delivered in the evening.

Validation tests were also carried out in the test environment in the evening. The patch was confirmed; the exploit no longer crashed patched hypervisors, and QA tests passed with the patched kernel.

The crisis unit moved towards monitoring the deployment and was then managed by the NOC (Network Operations Centre), which assumed the role of operational coordinator: monitoring overall progress, prioritising between regions and services, and maintaining the overall vision. Execution was handled by the Public Cloud and VPS experts, who performed reboots, live migrations, and anti-affinity constraints. The separation between coordination and execution was deliberate: the NOC coordinated while the operational teams acted and reported the metrics and technical events necessary for coordination.

Synchronisation with Customer Support (real-time progress status to respond to affected customers) and Security Teams (to monitor vulnerability and validate scope and end-of-operation criteria) was continuously ensured.

The operation ran under a “follow the sun” model, so the unit worked 24/7 with a rotation through different geographical areas. Three synchronisation points were held each day, covering the zone transitions. These brought together the NOC, operational experts, Support, and Security to: share the progress status by region, hand off between zones (what worked, procedural adjustments based on field feedback), and arbitrate the priorities for the next sequence.

Through a 24/7 rotation, the complete unit brought together the following teams: Kernel & Virtualisation (patch analysis, backport, validation), VPS and Public Cloud (deployment), NOC (management), Run & SRE (orchestration, anti-affinity, live migration), Datacentre Operations (hardware interventions), Customer Support (customer requests), Security (monitoring, scope, end of operation) and Communication (transparency, targeted notifications).

The assigned objective was clear: to patch and reboot the affected hosts as quickly as possible to reduce the vulnerability window, while minimising the impact on services. The main constraint related to scale: there were tens of thousands of machines to process, across all continents, knowing that case-by-case handling was materially impossible at this volume.

The main risk at this stage was the exploitation of the vulnerability, leading to a crash of the unpatched host. The crisis unit decided to apply the patch upstream of the reboot phase globally. In the event of a vulnerability exploit, a host crash would trigger its reboot and automatically apply the patch. Furthermore, there was a risk that the CVE could be exploited for malicious host control. No exploit code is publicly available, but we knew it was only a matter of time before a researcher succeeds in using the vulnerability to take control of the host. With this scenario being catastrophic, we knew that every minute counted.

A confident decision: unilateral patching with controlled impact

The Executive Committee approved the Go/No-Go in the evening: the first region was processed the following morning. For this number of machines, there is no scenario without impact. Negotiating a maintenance window with each customer, verifying each dependency, orchestrating each custom reboot: given the security risk, all of these steps were materially incompatible with an acceptable lead time.

The crisis unit therefore made a considered decision: unilateral patching with controlled impact, applied without waiting for the individual agreement of each customer, with the understanding that some services would experience an interruption. There were three reasons for this decision:

• not patching exposes the entire fleet to a high severity flaw
• case-by-case processing would lengthen delays and leave the majority of hosts vulnerable for weeks
• quick and comprehensive action protects the greatest number of people, even if it temporarily affects a minority

The priority was no longer to avoid the impact, but to minimise it, dilute it, and make it predictable. It is this stance that structured the entire operation: follow the sun, prioritisation of regions and anti-affinity planning.

Wednesday 8 July: start from Sydney

The choice of Sydney to test the deployment was simple: the number of hosts is limited, and the deployment range in local HNO (during the night) corresponds to the office hours of the teams in Europe. Starting with the easternmost region allowed us to:

• operate in the least busy area
• validate the procedure under real conditions, on a reduced scale, before industrialisation
• collect initial feedback before launching the European and North American regions.

The first waves of patch + reboot were applied to VPS hosts in Australia. The SYD2 region was completed at the beginning of the afternoon (Paris time), without incident. The procedures were adapted based on feedback from the field.

As soon as the procedure was stabilised, the follow the sun approach began: each region took over when it was their turn, in the morning in its local time, while transmitting the context to the next one. The first European wave (RBX, GRA6, WAW, DE, SBG, MIL, UK) is launched that same evening, at 6:30 PM Paris time.

Two scopes, two exposures: VPS first, Public Cloud later

The deployment was not uniform, as VPSs and Public Cloud differ in their architecture and customer exposure. The number of virtual machines on VPS hosts is higher and many companies and individuals use VPSs for test infrastructures, meaning that the probability of a customer testing the operating code on their virtual machine was very high and the impact was also high due to the number of virtual machines on each host.

For VPSs, the scope per host was contained and the customer impact for each reboot remained under control. Batches could be chained quickly, allowing a large portion of the fleet to be secured within the first 24 hours.

The Public Cloud presented a different kind of exposure. One region has thousands of customers, and one host can host critical instances. The most important regions have hundreds or even thousands of hosts with complex virtual customer infrastructures. We decided to prioritise:

• by region size: regions with lower density were processed first, in order to validate the robustness of the procedure at scale
• by number of exposed customers: high volume regions were orchestrated with higher granularity, batch by batch, to dilute risk

Stop thresholds and tempo control

Each wave of reboots was subject to a stop threshold: if the number of simultaneous failed hosts exceeded a defined threshold, the wave was suspended. This threshold was set at 15 hosts for high density regions (GRA, RBX, BHS) and at 5 hosts for others. A stop was also triggered at 06:00 AM or upon request from the local datacentre.

This mechanism prevented a hardware failure situation from worsening by continuing to reboot hosts that the DC technicians had not yet been able to process. It introduced a point of regulation between software automation and the physical reality of the field.

Not rebooting two instances of the same project at the same time: anti-affinity as a safeguard

The main risk for our customers during an operation like this is not the reboot itself, but the simultaneous interruption of several instances of the same project ensuring application resilience capable of handling a provider failure. A customer who has spread their workloads across multiple hosts to ensure high availability should not see all of their instances failover at the same time. The decision was made to go beyond compliance with the anti-affinity rules that may have been set out in customer deployments.

Therefore, for each customer project with instances spread across multiple hosts, our orchestrators calculated a colocation graph. Two hosts carrying instances of the same project were never rebooted in the same window: mutually exclusive waves were defined, and a host had to be back online before the next one in the same anti-affinity class could be started.

This anti-affinity was applied on a “best efforts” basis: it was respected in the majority of cases, but could not be 100% guaranteed across the entire fleet. The goal remained to sequence the impact so that it was absorbable on the application side. Customers whose entire operation relied on a single host would experience an occasional interruption, the window of which was announced.

Live migration of priority services and sensitive workloads

Behind each customer instance are controllers, APIs, data schemas, and internal databases. An uncontrolled reboot of the hosts running these services would create deadlocks: an unavailable internal service blocks further reboots, which blocks patching. What’s more, some OVHcloud services are based on virtual machines hosted on Public Cloud instances. Taking these cases into account is essential to limit customer impacts.

To avoid this cascade, the priority was reversed: a thorough analysis of the internal service dependencies was made upstream of each region reboot. Some internal services were live migrated. Their virtual machines were hot migrated to already patched hosts, the dependency chain was kept available, and then the host was released for reboot.

This procedure is long and consumes intensive hardware and human resources. It had to be limited to a small number of virtual machines to meet the objective of containing the migration over time.

Some workloads required special attention: in particular, Cloud Database services (DBaaS), internal Datalake, and Observability functions were migrated one VM at a time, without shutting down more than one machine at the same time, by delaying the update of their hosts for as long as possible. This smooth migration avoided cascading failures of services and ensured that the tools supporting operations remained available.

Technical incidents and adjustments during operations

An operation of this magnitude is never completed incident-free. Several technical issues were encountered and resolved during the campaign.

VMs not restarting after host reboot

The first major incident occurred during the first European wave: some virtual machines did not restart after their hypervisor was rebooted. Nova compute was reporting “Instance shutdown by itself” without synchronisation. The root cause was identified on day two: the libvirt-guests service was conflicting with nova compute and stopping instances on reboot without API-side sync. The fix applied was to disable and hide libvirt-guests.service on hosts before the reboot. With this fix, automatic VM reboots worked.

Data corruption on synchronous services

On the evening of the second day, the internal monitoring system reported corrupted data on several VMs spread across three clusters. The likely cause: the forced reboot occurred in the middle of a disk write. The graceful shutdown was therefore extended to 60 seconds before forced kill, to allow time for writes to complete. An automatic restart script for VMs that had been turned off was implemented.

Deadlock API in Paris

On the night of the second to the third day, the Nova and Neutron APIs in Paris entered a mutual deadlock: the Neutron API (limited to 10 processes) became saturated under a surge of requests from Nova, returning HTTP 503 for about two hours. The fix involved increasing the Neutron workers from 10 to 30 and the Apache processes from 10 to 32. The Paris and Milan B zones were delayed to allow time to stabilise.

Saturation support in BHS

On the BHS (Canada) site, API traffic reached 10 times the usual maximum traffic, saturating the Manager and support. Some customers discovered the impact before they received the communication. This case is a concrete example of the chain reactions within the infrastructure.

These bugs and encountered problems have all been managed within the operations framework, but will be taken into account to implement appropriate improvements on a permanent basis.

Reboots and hardware interventions

A reboot of tens of thousands of machines also involves a hardware dimension. There is a failure rate inherent in any server reboot. On the first night, approximately 20 to 30 hosts out of 6,000 did not come back on their own, with failing memory sticks, BIOS configuration, and inactive network interface. In the US, several hosts required a CMOS battery removal and a power drain before repower – a recurring hardware profile.

Each site had datacentre technicians deployed during the campaign. Their role was to:

• take emergency action on hosts that failed to reboot as signalled by the orchestrators
• replace faulty components (disks, memory sticks, power supplies)
• perform proximity gestures that no tool can automate: physical hard reboot checking lights, rack-related tasks

The datacentre technicians operated in priority intervention mode on failing hosts, in coordination with the Run/SRE teams that prioritised according to the customer exposure of the host. A host carrying critical instances that did not come back online was handled before a vacant host. This cross-prioritisation – software and physical – is what kept the tempo of the operation going.

Communication and support: informing affected customers

A unilateral patching with controlled impact cannot be created without a proportionate communications effort.

Targeted and progressive communication

Given the follow the sun approach, a global and undifferentiated communication would have made no sense. The chosen strategy was a targeted and gradual communication: addressed only to customers whose instances are hosted on the hosts scheduled for reboot, and triggered as the operation progressed, region by region, wave by wave.

Initially, the decision was made not to activate a public status page, to avoid exposing the deployment sequence. Communications took place through targeted notifications via our Support portal, with emails sent to customers with Business and Enterprise Support levels.

Finding: some messages were not delivered

Communication tools have technical limitations. For high-volume regions like GRA6 (almost 90,000 non-contacted customers), mass email sending is avoided to prevent a proliferation of support tickets.

In light of this, a pivot was decided on the second day: a conditional banner was implemented in the Manager using feature flipping. If the logged-in user is on the list of affected customer accounts (NICs), an informational message was displayed. Development was carried out during the day and the banner was deployed on the third day. A Public Cloud status page was also created at this time.

Despite these mechanisms, some messages did not reach their recipients, due to outdated contact addresses, filtered notifications, and delays between the scheduled reboot time and the sending time. Customers discovered the impact without prior warning. These points of failure have been identified as priority areas for improvement following the initial mitigation plan.

Support as a last-resort channel

For uninformed customers, or those requesting details, customer support was strengthened and prepared, with briefing upstream on the context of the operation, real-time access to the progress status by region and by host, and accelerated processing of tickets related to the campaign via a dedicated channel.

Support caught what the automated notifications could not carry. Rather than replacing communication, it compensated for some of its shortcomings.

Timeline of the operation

Insights

CVE-2026-53359 put our customers and our infrastructure at risk. The mitigation action plan resulted in a customer impact – it was contained, sequenced, announced, but still real. Communicating more extensively during the implementation of the action plan, while the infrastructure was not patched, would have significantly increased the risk to our customers, prompting some of them to “test” the publicly available exploit.

Rebooting a global fleet with zero impact was not an achievable goal. The objective was therefore the smallest impact compatible with the safety of the entire fleet.

Patching and restarting all Public Cloud hosts and VPSs had never been done within this time constraint. This emergency procedure was put in place in view of the risk related to the vulnerability. Previous cases have always been handled by a gradual reboot using the natural rotation rate of virtual machines on the infrastructure, combined with ad-hoc live migrations planned over a long time.

The teams involved in this operation have achieved an exploit with a very reasonable number of failures and customer impacts in proportion to the size of the project. However, as we are aware that the coming months could lead to further kernel vulnerability publications, it seems clear that this emergency procedure will have to be repeated. We will have to do better next time, both in controlling the gross impact of restarts, in informing customers beforehand and in the procedure for supporting affected customers during operations. Therefore, in the coming days and weeks, we will run a post-mortem process to identify the main impacts with our customers who experienced complex failures, and work on the problems encountered during internal operations, with the aim of improving our procedures in the future.