.png)
Reliable and well integrated into OpenStack, Ceph has been our Block Storage solution for years. But in order to offer more performance at a lower cost, we acquired Exten, and are now able to build our own NVMe block storage solution in C++ and Go. Our building goals are to:
- Use NVMe drives to offer high-performance volumes
- Use smart encoding algorithms to require less space on drives (Reed-Solomon encoding, compression)
- Keep clusters small and easy to maintain
Exten has now been running in production for more than a year. In this blog, we detail how it works, and what its architecture will open up next.
Click on the link at the end to give your feedback, and help shape Exten’s future.
Anatomy of a cluster
A typical Exten cluster is made of six servers. These servers are close to each other so that the latency between them is low, but not too close to limit the chances of multiple servers failing together. Each server has dozens of NVMe drives to store user data.
Exten is built around the NVMe protocol. It uses NVMe drives, but it also uses the NVMe over Fabric protocol to expose user volumes and to communicate between hosts. Volumes are virtual drives that customers can attach to their VMs. This protocol uses the same commands as NVMe drives, but over a network link (TCP or RDMA), and it's designed for high performance and low latency.
An advantage of NVMe over Fabric is that the Linux kernel supports it natively. Connecting an Exten volume to a Linux host is as simple as running an nvme connect command with the right parameters: the volume then appears just like any other local NVMe drive. As a Public Cloud customer, you don't see any of this — the volume is just attached to your instance, exactly like with Ceph.
Exten is split into a data plane written in C++ and a control plane written in Go. Both communicate through gRPC. The data plane receives IO through NVMe over Fabric and routes it to the NVMe drives. The control plane provides an API to create and delete volumes and to manage the cluster.
How to keep data safe
Keeping customer data safe is an Exten cluster's biggest responsibility, and we handle it at multiple levels.
On the control plane side, we use the well-known Raft Consensus Algorithm to guarantee the cluster's integrity in a fault-tolerant manner. It works by electing a leader and logging any new information — a new volume created, a drive enabled — in a log replicated across every host. If a host fails, the others keep working. The only limit is that a majority of hosts must stay up to avoid “split-brain” issues, so a six-node cluster can lose two hosts and keep working.
User data itself, or the actual IO done to the volumes, doesn’t go through Raft; that would be far too slow. Instead, we use another well-known algorithm: a Reed-Solomon code.
This error-correction code is conceptually simple. Take RS 4+2: we split data into four equal parts, then compute two new parts of the same size. If you can read any four of those six parts, you can recover the original data: the code lets you compute the two missing ones. That gives a ratio of 1.5 (data stored / data received), so we can lose two drives without losing any data. It's a highly efficient way to add redundancy. The naive alternative — usually called mirroring, which is just copying the whole data multiple times to different drives — would need three copies to survive losing two drives. With Reed-Solomon, we store half as much data for the same redundancy level.
Failing drives may be detected in two ways : either a read simply fails, or it returns with corrupted data — , which is detected thanks to a checksum we add to each sector. When a drive fails, an automatic process recomputes the missing part and stores it on a healthy drive, lowering the risk of data loss.
Doing I/O fast
With CPUs and RAM working faster, storage has to follow. Exten's goal is to make the most of the hardware it runs on and deliver high-performance IO to customers. Choosing the NVMe over Fabric protocol is a first step toward IO with as little overhead as possible, but the whole software stack is designed for performance.
The processing model is data-centric. At the Network Interface Card (NIC) level, we receive and send data through a list of fixed-size buffers stored in RAM. A given CPU processes each buffer with two goals: avoiding data copies and avoiding context switches.
To avoid data copies, the code manipulates pre-allocated buffers through pointers all the way from the NIC to the drive, and we only modify data when unavoidable (compression, Reed-Solomon encoding...). To avoid context switches, we handle IO through fully asynchronous methods that work on small pieces of data. A given CPU does as much processing as possible on a buffer before moving to the next, but it's a balance: the next buffer can't be delayed too long, and processing has to remain fair. We built a whole C++ framework tailored to these needs to keep the code clear and maintainable.
Beyond the code itself, several other factors help improve performance:
- the choice of the compression algorithm (no compression is sometimes the best choice, for example on encrypted traffic)
- the choice of the network protocol: RDMA is faster than TCP, but implies more deployment constraints
- the way we allocate CPU cores to volumes
On our Public Cloud offers, we also apply QoS rules to ensure fairness across all volumes.
What's next for Exten
Exten is now in production, and it provides volumes to Public Cloud customers today. It's transparent to customers and currently backs the same offers as Ceph — but it's a software stack that we fully control, built around NVMe for high performance.
We're constantly improving it, aiming to offer higher-performance volumes, more flexibility, and more reliability to customers. We're, for example, exploring the use of RDMA to improve performance (we currently use TCP).
The first step of the Exten journey was to provide a reliable offer replacing Ceph. Now that we fully control the stack, we're ready to add new features and adapt the offer to customers' needs.
Do you have Block Storage use cases we should know about, or features you feel are missing? Tell us about them on Discord or via or on the cloud mailing-list: your feedback shapes where Exten goes next.