A look back at a remediation operation carried out in one week across our entire KVM fleet, following a patching strategy whose central goal was to minimize the impact on our customers’ services.

A vulnerability in the virtualization subsystem

On Tuesday, July 7, in the early afternoon, a security alert came in regarding CVE-2026-53359, a use-after-free vulnerability affecting the shadow-paging subsystem of KVM x86 in the Linux kernel. The flaw, several years old, was disclosed publicly on July 6; blog posts from other cloud providers started appearing as early as the 7th.

KVM is the virtualization engine underlying the vast majority of instances hosted at OVHcloud. The mechanism works as follows: when an external modification to a Page Directory Entry (PDE) occurs, the RMAP entry can retain a reference to a memory page that has already been freed. The kernel then dereferences this stale page, which can cause the hypervisor to crash or, in worst-case scenarios, lead to privilege escalation on the host side. The exploit is reproducible: an internal test on an unpatched host triggers a crash in about two minutes.

Every x86 Linux kernel prior to the fix commit is affected, regardless of distribution. The official patch was backported onto our Debian production kernels.

Three main risks had to be considered:

• A VPS customer uses the exploit to crash the host running their virtual machine: an uncontrolled crash and reboot of several hundred customer virtual machines.
• The same scenario on Public Cloud instances has a similar impact, but the number of virtual machines per host is smaller and those machines are more powerful and used for more sensitive systems (databases, queue managers, load balancers, etc.). These systems are often used in composite application architectures with complex dependencies.
• The likelihood of an upcoming release of a host-takeover exploit, which would immediately raise the risk to an unacceptable level given the impact on customer data confidentiality and infrastructure integrity.

For OVHcloud, the scope represented tens of thousands of hypervisor host servers, hosting around one million virtual machines. The question therefore wasn’t whether to patch, but how to carry out this operation at such a scale, knowing that zero customer impact was not achievable.

The options on the table for mitigating the risk

1: Wait for the official patched kernels

This option made us dependent on a third-party timeline, leaving us in a risk state for an uncontrolled period. We ruled it out quickly given the risk involved.

2: Live patch

Applying a live patch requires enabling this type of operation in the kernel configuration, which by design allows its behavior to be modified on the fly — reducing its hardening level and limiting detection capabilities in the event of a compromise. Moreover, live patching is inherently a sensitive procedure that can lead to fleet-wide instability. It’s an option that buys time while waiting for a lasting solution. We decided not to make this trade-off, since it would have significantly increased the risk in the event a host-takeover exploit was published.

3: Mitigation by disabling nested virtualization

Disabling nested virtualization on hosts renders the exploit inoperative. We have no visibility into how our customers use this feature, so it’s impossible to determine the impact on customer services, and this feature is required to retain the ability to live-migrate an instance from one physical host to another. We ruled out this path quickly.

4: Live migrate

Organize a live migration of virtual machines from vulnerable hosts to empty hosts that have been patched in the meantime. This option is very satisfying in terms of service continuity, since the migration happens without impact on the virtual machines, aside from reduced performance during the migration. This option takes a great deal of time due to the copying of virtual machines from host to host; it isn’t realistically usable at fleet scale given the goal of protecting our customers within days rather than months. We decided to keep this option for certain critical virtual machines only, since each live migration causes a significant delay in batch processing.

5: Backport the patch onto our kernels and reboot all hosts

This is ultimately the option that was chosen, and it is detailed in the rest of this article.

Tuesday afternoon: activating and organizing the crisis cell

As soon as the vulnerability was confirmed, the priority was to build a structured, coordinated response; the analysts handling the initial analysis very quickly understood the stakes for the days ahead. The information was shared internally in the early afternoon. Several coordination spaces were opened: one for technical coordination, one for crisis coordination, one for US operations, and one for customer communication and support. At the same time, the Kernel teams prepared and backported the fix; the first patched kernel was delivered that evening.

Validation testing was carried out in a test environment that evening. The patch was confirmed: the exploit no longer crashed patched hypervisors, and QA tests passed with the patched kernel.

The crisis cell evolved into a rollout-tracking function and was then led by the NOC (Network Operations Center), which took on the role of operational coordinator: tracking overall progress, arbitrating priorities between regions and services, and maintaining the big picture. Execution was carried out by the Public Cloud and VPS experts, who handled reboots, live migrations, and anti-affinity constraints. The separation between coordination and execution was deliberate: the NOC coordinates, while operational teams act and report back the metrics and technical events needed for coordination.

Synchronization with Customer Support (real-time status updates to respond to affected customers) and the Security teams (monitoring the vulnerability, validating scope and end-of-operation criteria) was maintained continuously.

Since the operation followed the sun, the cell operated 24/7 with rotation by geographic zone. Three sync points were held each day, covering zone handoffs. These brought together the NOC, operational experts, Support, and Security to: share progress by region, hand off context between zones (what worked, procedural adjustments based on field feedback), and set priorities for the next sequence.

The full cell brought together, in 24/7 rotation: the Kernel & Virtualization teams (patch analysis, backporting, validation), VPS and Public Cloud (deployment), NOC (steering), Run & SRE (orchestration, anti-affinity, live migration), Datacenter Operations (hardware interventions), Customer Support (customer requests), Security (monitoring, scope, operation closure), and Communications (transparency, targeted notifications).

The assigned objective was clear: patch and reboot the affected hosts as quickly as possible to reduce the vulnerability window, while minimizing the impact on services. The main constraint was scale: tens of thousands of machines to process, across every continent, knowing that case-by-case handling was materially impossible at this volume.

The main risk at this stage was exploitation of the vulnerability, causing an unpatched host to crash. The crisis cell decided to apply the patch fleet-wide ahead of the reboot phase. In the event the vulnerability was exploited, a host crash would trigger its reboot and automatically pick up the patch. In addition, there was a risk that the CVE could be exploited for malicious host takeover. No exploit code was publicly available, but it was understood to be only a matter of time before a researcher managed to use the vulnerability to take control of a host. Since this scenario would be catastrophic, every minute counted.

UA deliberate decision: unilateral patching with controlled impact

The Executive Committee approved the Go/No-Go that evening: the first region would be handled the next morning. At this number of machines, no zero-impact scenario existed. Negotiating a maintenance window with each customer, checking every dependency, orchestrating every reboot individually — all of these steps were materially incompatible with a timeline acceptable given the security risk.

The crisis cell therefore made a deliberate decision: unilateral patching with controlled impact, applied without waiting for each customer’s individual consent, knowing that some services would experience an interruption. The reasoning rested on three points:

• not patching exposes the entire fleet to a high-severity flaw;
• case-by-case handling would extend the timeline and leave most hosts vulnerable for weeks;
• fast, global action protects the largest number of customers, even if it temporarily impacts a minority.

The priority was no longer to avoid impact, but to minimize it, spread it out, and make it predictable. This is the principle that structured the entire operation: follow the sun, regional prioritization, and anti-affinity planning.

Wednesday, July 8: starting from Sydney

Choosing Sydney to test the rollout was straightforward: the number of hosts is limited, and the local off-peak deployment window (overnight) aligns with office hours for teams in Europe. Starting with the easternmost region allowed:

• operating in the least-loaded zone;
• validating the procedure under real conditions, at reduced scale, before industrializing it;
• gathering initial feedback before launching the European and North American regions.

The first waves of patch + reboot were applied to VPS hosts in Australia. The SYD2 region was completed in the early afternoon (Paris time), without incident. Procedures were adjusted based on field feedback.

As soon as the procedure stabilized, follow the sun kicked in: each region took over in turn, during its local morning, passing context on to the next. The first European wave (RBX, GRA6, WAW, DE, SBG, MIL, UK) was launched that same evening, at 6:30 PM Paris time.

Two scopes, two exposures: VPS first, Public Cloud second

The rollout wasn’t uniform. VPS and Public Cloud differ both in architecture and in customer exposure. The number of virtual machines per VPS host is higher, and many businesses and individuals use VPS for test infrastructure; the likelihood of a customer testing the exploit code on their own virtual machine is very high, and so is the impact, given the number of virtual machines on each host.

On VPS, the scope per host is contained and the customer impact per reboot stays manageable. Batches could be chained quickly, which secured a large part of the fleet within the first 24 hours.

Public Cloud presents a different order of exposure. A single region concentrates thousands of customers, and one host can hold critical instances. The largest regions have hundreds or even thousands of hosts with complex virtual customer infrastructures. We decided to prioritize:

• by region size: lower-density regions were handled first, to validate the robustness of the procedure at scale;
• by number of exposed customers: high-volume regions were orchestrated with finer granularity, batch by batch, to spread out the risk.

Stop thresholds and pace control

Each wave of reboots was subject to a stop threshold: if the number of simultaneously down hosts exceeded a defined limit, the wave was paused. This threshold was set at 15 hosts for high-density regions (GRA, RBX, BHS) and 5 hosts for others. A stop was also triggered at 6:00 AM, or on request from the local datacenter.

This mechanism prevented worsening a hardware-failure situation by continuing to reboot hosts that DC technicians hadn’t yet been able to address. It introduced a regulation point between software automation and physical, on-the-ground reality.

Never rebooting two instances of the same project simultaneously: anti-affinity as a safeguard

The main risk for our customers in an operation like this isn’t the reboot itself — it’s the simultaneous interruption of multiple instances of the same project meant to guarantee application resilience against a provider failure. A customer who has spread their workloads across several hosts to ensure high availability must not see all of their instances go down at once. The decision was made to go beyond the anti-affinity rules that may have been defined in customer deployments.

So, for every customer project with instances spread across multiple hosts, our orchestrators compute a co-location graph. At no point are two hosts carrying instances from the same project rebooted within the same window: mutually exclusive waves are defined, and a host must be back in service before the next one in the same anti-affinity class is launched.

This anti-affinity was applied on a best-effort basis: it was respected in the majority of cases, but couldn’t be guaranteed 100% across the entire fleet. The goal remained to sequence the impact so that it could be absorbed at the application level. Customers whose instances all sit on a single host experienced a one-off interruption, with an announced window.

Priority live migration of sensitive services and workloads

Behind every customer instance sit controllers, APIs, data planes, and internal databases. An uncontrolled reboot of the hosts carrying these services would create deadlocks: an unavailable internal service blocks the rest of the reboots, which blocks the patching. In addition, some OVHcloud services rely on virtual machines hosted on Public Cloud instances. Accounting for these cases was essential to limit customer impact.

To avoid this cascade, priority was reversed: an in-depth analysis of internal service dependencies was carried out ahead of each region’s reboot. Some internal services were live-migrated. Their virtual machines were moved live to already-patched hosts, the dependency chain was kept available, and the host was then freed up for reboot.

This procedure is lengthy and intensively consumes hardware and human resources. It had to be limited to a small number of virtual machines to keep the migration within the operation’s time constraints.

Certain workloads required particular attention: in particular, Cloud Databases (DBaaS), the internal Datalake, and Observability functions were migrated one VM at a time, never shutting down more than one machine simultaneously, deferring the update of their hosts for as long as possible. This gentle migration approach avoided cascading failures on these services and ensured that tools supporting operations stayed available.

Technical incidents and adjustments during the operation

An operation of this scale doesn’t run without incidents. Several technical issues were encountered and resolved during the campaign.

VMs not restarting after host reboot

The first major incident appeared as early as the first European wave: virtual machines weren’t restarting after their hypervisor rebooted. Nova compute reported “Instance shutdown by itself” without synchronization. The root cause was identified on the second day: the libvirt-guests service was conflicting with Nova compute and stopping instances on reboot without syncing with the API. The fix applied was to disable and mask libvirt-guests.service on hosts before reboot. From that fix onward, automatic VM restarts worked correctly.

Data corruption on synchronous services

On the evening of the second day, the internal monitoring system reported corrupted data on several VMs spread across 3 clusters. The probable cause: the forced reboot occurred in the middle of a disk write. The graceful shutdown window was then extended to 60 seconds before a forced kill, to give writes time to complete. An automated restart script was put in place for VMs that had remained down.

API deadlock in Paris

On the night between the second and third day, the Nova and Neutron APIs in Paris entered a mutual deadlock: the Neutron API (capped at 10 processes) became saturated under a burst of requests from Nova, returning HTTP 503 errors for about two hours. The fix consisted of increasing Neutron workers from 10 to 30 and Apache processes from 10 to 32. Paris and Milan’s Zone B were postponed while things stabilized.

Support saturation in BHS

At the BHS site (Canada), API traffic reached 10 times the usual peak, saturating the Manager and support. Some customers discovered the impact before receiving communication about it. This case is a concrete illustration of chain reactions within the infrastructure.

These bugs and issues were all handled within the operation, but will be factored into lasting improvements going forward.

Reboots and hardware interventions

Rebooting tens of thousands of machines also has a hardware dimension. There’s an inherent failure rate to any server reboot. On the first night, roughly 20 to 30 hosts out of 6,000 didn’t come back on their own: failed memory sticks, BIOS configuration issues, inactive network interfaces. In the United States, several hosts required a CMOS battery removal and a power drain before repowering — a recurring hardware profile.

Every site had datacenter technicians mobilized as reinforcement during the campaign. Their role:

• respond urgently to hosts flagged by orchestrators as failing to reboot;
• replace faulty parts (disks, memory sticks, power supplies);
• carry out hands-on tasks that no tool can automate: physical hard reboots, checking indicator lights, working directly in the rack.

Datacenter technicians operated in priority-response mode for failed hosts, coordinating with the Run/SRE teams, who prioritized based on a host’s customer exposure. A host carrying critical instances that failed to come back took priority over a vacant host. This cross-prioritization — software and physical — is what kept the operation’s pace.

Communication and support: informing affected customers

Unilateral patching with controlled impact can’t be conceived of without a proportionate communication effort.

Targeted, progressive communication

Given the follow-the-sun dynamic, a broad, undifferentiated communication would have made no sense. The strategy adopted was targeted, progressive communication: sent only to customers whose instances were hosted on hosts scheduled for reboot, and triggered progressively as the operation advanced, region by region, wave by wave.

Initially, the decision was made not to activate a public status page, so as not to expose the deployment sequence. Communications went out through targeted messages via our Support portal, with emails sent to customers with Business and Enterprise Support levels.

Finding: some messages weren’t delivered

Communication tools have technical limits. For high-volume regions like GRA6 (nearly 90,000 uncontacted customers), mass emailing was ruled out to avoid a flood of support tickets.

Faced with this, a pivot was decided on the second day: implementing a conditional banner in the Manager via feature flipping — if the logged-in user appears on the list of affected customer accounts (NICs), an information message is displayed. Development was carried out within the day, and the banner was deployed on the third day. A Public Cloud status page was also created at that point.

Despite these measures, some messages didn’t reach their recipients: outdated contact addresses, filtered notifications, a mismatch between the scheduled reboot time and the send time. Some customers discovered the impact without having been warned beforehand. These breakdown points were identified as priority areas for improvement following the initial mitigation plan.

Support as the last-resort channel

For customers who weren’t informed, or who wanted more details, customer support was reinforced and prepared: briefed in advance on the operation’s context, given real-time access to progress by region and by host, and given accelerated handling of campaign-related tickets through a dedicated track.

Support caught what automated notifications couldn’t carry. It doesn’t replace communication — it makes up for some of its gaps.

Operation timeline

When	Action
Tuesday 7/07 afternoon	Alert received on CVE-2026-53359. Coordination spaces opened. Kernel patch prepared and backported.
Tuesday evening	Validation testing in the lab (patch confirmed, exploit reproduced). Executive Go/No-Go: first region the next day. Decision to pursue unilateral patching with controlled impact.
Wednesday 8/07 morning (Sydney)	First waves of patch + reboot on VPS SYD2 hosts. Runbook validated. SYD2 completed without incident.
Wednesday 8/07 evening	First European VPS wave launched (RBX, GRA6, WAW, DE, SBG, MIL, UK). Public Cloud starts (GRA1, SGP1, SYD1, AP-SOUTHEAST-SYD-2).
Thursday 9/07	libvirt-guests bug identified (VMs not restarting) → fix applied. DBaaS live migration (4,300+ VMs). BHS support saturation (10x traffic).
Thursday 9/07 evening → Friday 10/07	Wave 3 (SBG8, GRA4, GRA8, Public Cloud BHS1). Paris API deadlock → Paris/Milan AZ-B postponed. Communication pivot: Manager banner deployed. Public Cloud status page created.
Friday 10/07 → Sunday 12/07	Continued VPS + Public Cloud rollout region by region, follow the sun, anti-affinity. GRA3 extended past the hard stop (Axel’s validation). DE1/SBG5 continue through Sunday night.
Monday 13/07 → Sunday 19/07	Final regions (BHS5, GRA7, GRA9, GRA11, SBG7, MIL AZ-A, UK1, US-EAST-VA-1). Remaining hosts live-migrated to avoid further maintenance windows.
End of operation (D+11)	All machines patched. Customer impact contained and sequenced.

Looking ahead

CVE-2026-53359 put our customers and our infrastructure at risk. The mitigation action plan led to customer impact — contained, sequenced, and announced, but real. Communicating in more detail while the action plan was being executed, as long as the infrastructure remained unpatched, would have significantly increased the risk to our customers, by encouraging some of them to “test” the publicly available exploit.

Rebooting a fleet of this global scale with zero impact was not an achievable goal. The objective was therefore the smallest possible impact compatible with the security of the entire fleet.

Patching and restarting every Public Cloud and VPS host had never been done before under this kind of time pressure. This emergency procedure was put in place in light of the risk posed by the vulnerability. Previous cases had always been handled through a gradual reboot, relying on the natural turnover rate of virtual machines across the infrastructure, combined with ad-hoc live migrations planned over a longer timeframe.

The teams involved in this operation achieved a result with a very reasonable number of failures and customer impact, proportional to the scale of the undertaking. However, aware that the coming months could bring further kernel vulnerability disclosures, it seems clear that this emergency procedure will need to be repeated. We’ll need to do better next time, both in controlling the raw impact of the reboots and in informing customers ahead of time and supporting affected customers during operations. In the days and weeks ahead, we will identify the main impacts on customers who experienced complex outages, and work through the issues encountered during operations internally, in a post-mortem, with the goal of improving our procedures going forward.