Microphotograph of the Motorola 68000 microprocessor die. Image courtesy of Wikimedia Commons, licensed under CC BY 3.0.
No changes were made.

Table of content:

• What is microcode?
• What is it used for?
• Benefits/Drawbacks
• Microcode Architecture
• Microcode Update Integration
• Microcode Update format
• Microcode Update Methods
• Microcode Update Methods Comparison
• Microcode Update challenges at OVHcloud
• OVHcloud Boot Process
• OVHcloud Microcode validation

What is microcode?

As Gordon Moore predicted in 1965, in what is now known as Moore’s Law, the number of transistors in an integrated circuit would double every two years. This exponential growth has driven the design of smaller, faster, and more efficient transistors, pushing CPUs to unprecedented levels of complexity. As a result, a more flexible way of converting machine code into CPU circuit-level operations is needed. Although MIT pioneered software-controlled circuitry in the 1950s, the widespread use of micro-programming for CPU was delayed for decades by the limited capacity and high cost of storage.

Microcode is an abstraction layer that sits between the CPU’s hardware and machine code (a binary representation of the compiled program). It translates machine instructions (basic steps a computer performs), state machine data, or other inputs into sequences of detailed circuit-level operations. It separates machine instructions from the control signal (i.e., electrical impulses), thereby enabling greater flexibility in designing and modifying instructions. Unlike hardwired instruction decoding, which requires hardware changes for bug fixes, microcode software-based updates provide a simpler, more efficient patching process.
Microcode software is exclusively edited by the hardware manufacturer, and is closely tied to the particular hardware it runs on. This means microcode is typically proprietary software (Intel x86, AMD x86), with the exception of a few open-source hardware CPUs.

What is it used for?

• Instruction decoding
• CPU bug fixes
• Exception handling
• Power management
• Complex CPU features

Benefits and drawbacks

Microcode architecture

Instructions specific to CPU architecture (or macro-instructions) are complex and must be broken down into sequences of simple instructions.

• Decoder: translates CPU architecture-specific instruction to micro-operation
  • Short Decoder + operation packing: widely used hardware-based decoder that can incorporate multiple short decoders
  • Long Decoder:  hardware-based decoder operating similarly to a more intricate set of instructions
  • Vector Decoder – microcode engine: software-based decoder for rare and very complex instructions
• Scheduler
  • Micro-ops are reordered and then fed into the pipeline
• Processing
  • Micro-ops are processed by the corresponding execution unit

Microcode update integration

Microcode uses 2 types of storage:

• Microcode ROM: for storing the program
• Microcode RAM: for storing the microcode update

Match registers are used to provide breakpoints in the microcode ROM

When a particular address is reached, it’s rerouted to the microcode RAM containing the update.

Since microcode patches are stored in a low-latency volatile on-chip RAM, it’s fairly limited in size and not persistent. Each time the system boots, it needs to:

• be in kernel mode (or supervisor mode)
• load microcode update into RAM
• write the update’s virtual address to the Microcode Specific Register (MSR)

By design, microcode patches add extra condition checks, which slows down CPU performance. Additionally, patch application isn’t always successful, and in some cases, only provides a partial fix.

Microcode update format

CPU manufacturers distribute microcode updates as bundled microcode files. A bundle microcode file is a concatenation of individual microcode files; each individual file is a single patch for a unique CPUID (the signature of a CPU hardware model within a product). Practically speaking, a microcode bundle contains all the microcode updates for every CPUID from a specific CPU manufacturer, as of a particular date. Each microcode file includes a header (see Intel structure table below and the payload containing the patch).

This payload consists of match registers (breakpoints) and triads (containing the microcode patch).

Triggers represent conditions under which control is transferred from microcode ROM to patch RAM. 

Each triad contains:

• 3 micro-ops: microcode instruction that runs code
• Sequence word: for redirecting control flow

Microcode update methods

Microcode updates can be applied at one of the following layers:

• Firmware layer
• Kernel layer
• Operating system layer

Regardless of the method used, the impact on CPU performance is consistent for all microcode updates.

Microcode update methods comparison

There are many ways to update a microcode. Here’s an extensive list of update options:

____

¹ Assuming the custom kernel is compatible with hardware

² Assuming the running OS is the one applying the microcode update

Microcode update challenges at OVHcloud

While CPU manufacturers strongly recommend installing the latest microcode updates, no contractual obligations compel end customers to install the most recent microcode versions. However, it might be required by internal policy and/or industry certifications and standards.

Microcode updates are usually distributed through system firmware (BIOS/UEFI) or OS patches, in partnership with hardware manufacturers and software editors.

As a server manufacturer, OVHcloud enables its customers to access these microcode updates.

Automating a diverse range of BIOS/UEFI editors and versions is a challenge for the company given its broad hardware range, strategic factors, and a geographically diverse supply chain. As a result, it doesn’t update microcode at the firmware layer.

Moreover, OVHcloud maintenance is limited to hardware and firmware layers, and doesn’t include accessing the operating system on the customer’s bare-metal server. So updating the microcode through an OS-dependent method is ruled out as an option.

OVHcloud boot process

As previously mentioned, the wide range of hardware motherboards makes it impossible to automate the boot mode switch from BIOS/UEFI. This is why OVHcloud uses a temporary, minimalist, in-memory OS for the boot process. One major advantage of this open-source boot firmware is its powerful scripting, which extends the capabilities of traditional Preboot Execution Environment (PXE) without the need for BIOS/UEFI reflashing. The boot process unfolds in the following steps:

Depending on whether the customer chooses to boot from disk, a rescue environment, or a custom iPXE script, the corresponding iPXE script will run.

If a custom iPXE script isn’t chosen, the system downloads the latest validated AMD or Intel (CPU-specific) microcode bundle and signature from OVHcloud. Once the signature is validated, the microcode patches are distributed across all the CPU’s cores. The system then boots from a local disk or a remote image.

If the customer decides to boot using an iPXE script, the API will generate the iPXE script as per the customer’s specifications, with no changes made. This means that without a custom iPXE script specifying otherwise, customers won’t receive microcode updates.

OVHcloud microcode validation

Whenever a CPU manufacturer releases a new microcode, OVHcloud repackages the microcode bundles (separate AMD and Intel versions). As expected, the cloud provider wants the ability to skip microcode patches for CPUs not in its infrastructure or those that have created issues on some of its CPUs problems. OVHcloud also enables the signing of bundles to ensure data integrity. Once the bundle microcode file is published to the file server, automated testing begins on dedicated servers, one for each affected CPU and platform.

The following actions are carried out on each testing server:

• server reboot from rescue with the microcode patch(s) to validate
• CPU microcode version check to validate iPXE script from the rescue boot
• disk(s) erasure to remove artefacts from previous OS re-installs and install a minimalist Linux distribution
• server reboot from disk and proper boot check
• CPU microcode version check to validate microcode update for boot from disk

Once validated, the new URL to the bundle microcode file is replaced in all the rescue and iPXE boot scripts; human intervention is required to manually change the URL. Depending on the severity, a targeted email can be sent to the affected customers. The new microcode patch only takes effect after the next server hardware reboot³. OVHcloud never reboots a dedicated server without the customer’s consent.

____

³ A (hard) reboot from the OVHcloud Control Panel or API is needed. Soft reboot (reboot command from the OS) doesn’t apply the microcode patch.

Sources

• Microcode (Wikipedia)
• Micro-operation (Wikipedia)
• Reverse Engineering x86 Processor Microcode (Usenix)
• CVE-2023-20593/Zenbleed (OVHcloud Blog)
• iPXE – open source boot firmware [buildcfg:image_ucode]
• MCExtractor (GitHub)
• The Linux Microcode Loader
• A quick overview of Netboot startup at OVHcloud

Today, we’re going to embark on a journey of discovery, and unveil our latest product: Bare Metal Pod.

You know us for the services we provide: bare metal servers, managed and unmanaged virtualisation platform, our 40+ public cloud services, domain names and telco.

This is just the tip of the iceberg, and to understand why we built and now offer Bare Metal Pod, we have to dig deeper.

So let’s begin this journey exploring the origins of Bare Metal Pod, and in later articles we’ll cover the more technical details—there’s a lot to touch on.

The OVHcloud way: more than just servers

As a cloud services provider, we supply the different platforms mentioned above. But most importantly, we have to take care of the infrastructure dedicated to these services, from the buildings, power and cooling to the software stack and automation required.

And we’ve been doing just this since 2001. It all started with the opening of our first datacentre in Paris, then building our own servers the next year, and our proprietary water-cooling solution the year after that.

At the core, we are all about efficiency, automation, and sustainability:

• Repurposing buildings as datacentres
• Designing our own servers to optimise performance and cost
• Maximising cooling efficiency to cut waste
• Automating everything to reduce errors and delays

And, in all modesty…. we’re pretty good at these.

Optimising datacentres like a pro

Basically, building our own servers in our Croix (FR) and Beauharnois (CA) plants means packing a ton of servers into a square metre. We’re talking about 4 custom racks, each hosting 48 servers, all in just 3 sq.m and using up to 160kW of 12V DC power. This gives us a server density of about 5000W per sq/ft, which beats out 90% of the industry.

And on top of that, we’ve got our proprietary water-cooling system—we save energy by not using AC for our servers. To further optimise air cooling, each of our rack is equipped with a large condenser (we call it a chilled door) at the rear of the rack, dissipating regular server heat in our water system. This keeps the datacentre comfortably warm for our staff and the network equipment, and extends hardware lifespan (less maintenance, fewer replacements, fewer outages….so more savings).

In addition to the physical optimisations we’ve just mentioned is our automation system. When a server or a cluster of servers have been assembled and tested in our plant, it’s sent to the datacentre, racked and connected to power, network, and water-cooling systems by our DC staff.

And from there, everything is automated. From server power management, discovery, testing, and readiness checks, to the moment it’s selected by a customer using their Control Panel, and then configured. No human interaction is required, meaning no delay and no error.

And these operations have been optimised and refined for over 20 years.

Enter Project Gold-o-rack

So in June 2023, a small team was assembled to review, analyse and build a new version of this system. We had 3 goals:

• Provide customers with dedicated on-premises autonomous racks
• Offer custom-built, plug-and-play Bare Metal Pods
• Upgrade the automation and security of our own datacentres

And that’s how Project Gold-o-rack came to be—a tribute to Goldorak (Grendizer), the legendary 70s anime mecha that crushed its enemies with style. Like its namesake, our system is powerful, autonomous, and unstoppable.

Using opensource technology was a must, as we absolutely can’t do without transparency and community support. So we went for OpenStack, Netbox, Grafana, and developed our own network management and automation system, and much more.

By September 2023—just three months later—we had a fully functional 24U rack, deployable and operational in 25 minutes. That’s not just fast—that’s insanely fast.

Security was a top priority since these racks would be installed in third-party datacentres. We quickly applied for SecNumCloud qualification, leveraging our existing compliance expertise.

Then, it hit us: why not offer this as a full-fledged product? And that’s how Bare Metal Pod came to be—dedicated, secure, and fully automated.

We structured the product into three key components:

1. On-Prem Cloud Platform (OPCP): The autonomous rack, with its own KMS and encryption mechanisms
2. Bare Metal Pod: Built on OPCP, hosted in our datacentres, and SecNumCloud-compliant
3. Cloud Store: A software catalogue enabling automated deployment within the rack

In June 2024, OPCP was ready, just 12 months after the 1st meeting… and shortly after we got the “green light” from the ANSSI, allowing us to pursue the SecNumCloud qualification process.

And if you were at, or watched our Summit Keynote in November 2024, you definitely saw it live…

What’s under the hood?

As an autonomous rack, it contains:

• Power Distribution Units
• Network equipment for internal and external connectivity
• Servers, including a Pod Controller

There are 9 Bare Metal server models available, from 16 to 256 cores, from 128 GB to 2.5 TB of memory, up to 792 TB NVMe SSD (RAW), Nvidia L4 and L40s GPU depending on your needs.

And the best part is that you can mix and match them, to build and manage the perfect autonomous rack, while keeping full control on security and resources.

We’ve got a total of 607 models in Bare Metal Pod, enough for nearly any configuration and need. And with up to 1500 servers in a single Pod, the possibilities are endless.

And on top of these servers, we are building an automated software library: the Cloud Store. Enclosed in the Bare Metal Pod, the Cloud Store will offer the Pod admin a selection of OS, virtualisation platforms and various software that can be pushed, installed, configured automatically on the servers in the Pod. This includes built-in security, monitoring, and logging integrated in the Pod monitoring tools.

And herein¹ lies the main challenge: making sure an entire collection of software from various editors can cohabit and interact with a single, opensource monitoring platform, a KMS, and an IAM without breaking anything…

Coming up next…

That’s a wrap for now! In the next article, we’ll deep-dive into hardware, networking, and security. Stay tuned!

Some of the Bare Metal servers options:

• Scale A1 – A8: Equipped with 4th Gen Intel Xeon Gold or AMD EPYC 9004 series processors, these servers provide between 16 to 256 cores and 128 GB to 1 TB of DDR5 ECC RAM. They are suitable for:
  • Hosting SaaS and PaaS solutions
  • Virtualisation
  • Database hosting
  • Containerisation and orchestration
  • Confidential computing
  • High-performance computing
• Scale-GPU 1 – 3: Featuring NVIDIA L4 GPU cards (x2 or x4) and up to 1.2 TB of DDR5 ECC RAM, these servers are ideal for:
  • 3D modelling
  • Media streaming
  • Virtual Desktop Infrastructure (VDI)
  • Data inference

• HGR-HCI I1 – I4: With dual 5th Gen Intel Xeon Gold or 4th Gen AMD EPYC 9004 series processors, these servers provide between 16 to 72 cores and up to 2.5 TB of DDR5 ECC RAM. They are suitable for:
  • Hyperconverged infrastructure
  • Virtualisation
  • Database hosting
  • Containerisation and orchestration
  • Confidential computing
  • High-performance computing
• HGR-SDS 1 – 2: Equipped with dual 5th Gen Intel Xeon Gold processors, these servers offer between 16 to 48 cores and up to 1.5 TB of DDR5 ECC RAM. They are ideal for:
  • Software-defined storage solutions
  • Object storage solutions
  • Big data
  • Database hosting
• HGR-STOR 1 – 2: Featuring a 5th Gen Intel Xeon Gold processor with 36 cores and up to 512 GB of DDR5 ECC RAM, these servers are designed for:
  • Archiving
  • Database hosting
  • Backup and disaster recovery plans
• HGR-AI-2: Equipped with NVIDIA L40s GPU cards (x2 or x4) and up to 2.3 TB of DDR5 ECC RAM, these servers are optimized for:
  • Machine learning
  • Deep learning

(And many other options… you get the idea.)

1. My editor liked the word and I found it cool too. https://www.collinsdictionary.com/dictionary/english/herein ↩︎