In parallel with this evolution, more and more uses cases involving publicly sharing data have arisen. The amount of data has exploded, in almost all applications, and IT people don’t want to manage storage anymore (maintenance, failure, scalling…).

As now, all the major languages have their object storage libraries (at least for S3 APIs), and tones of applications integrate by default the connectors to those services. And as many IT people, I switched my reflexes to think Object Storage first, and handle some exceptions where network file systems are a better solution.

When applications need storage performances…

During many years, performances were not the main focus. The goal was to take advantage of unlimited space and public access to store images, documents, code…

Nowadays, part of the applications need more speed because the objects are bigger and plentiful. That’s the case for Big Data use cases and IA scenarios but also for the majority of modern applications.

• Those who work on big data analysis know that you can wait hours before getting your result in many cases. Working on data to analyze users behavior, system deviation, cost optimization requires powered compute resources but also high performance storage. Big Data clusters needs to access the date quickly to provide a result in time.
• Training IA models and fine tuning the predictions requires a lot of data analysis also that takes hours and slow down your progress if your storage solution is the bottleneck.
• Serving media that are bigger and bigger (4K video, HD pictures on smartphones, …) on high speed network (fiber, 5G) challenge the previous solutions.

OVHcloud Standard Object Storage hosts petabytes of data across the world on a strong and resilient infrastructure but our classical offer was not design to address those use cases. The performances of our classical object storage where not enough in those situations.

A new Object Storage service for demanding applications

So what is high performances for an object storage service? What are we speaking about?

There are mainly two things to consider when you need speed on your objects. The first one is the API. This is the first component that answers the requests you’ll send to the cluster, so the simplest API call should react as a flash. That’s the assurance that all the engine and automation tools on the cluster are optimized and not consuming precious milliseconds. The second one is the bandwidth. You’ll expect that now your object is identified, it should be delivered to you as soon as possible using the maximum capacity of disks and network.

So we are excited to provide OVHcloud High Performance Object Storage that fit those requirements in the upcoming weeks. The general availability should be the 21rst February at first in Strasbourg, then mid-March in Gravelines.

Some numbers

We’ll make some tests and compare that new service with the existing service at OVHcloud and the market leader on Object Storage, AWS S3. We’ll do those tests from OVHcloud network on a Public Cloud instance B2-120 which have a 10G public interface. This instance is located in Gravelines (north of France). The three tested services are located in central Europe, Strasbourg (France) for OVHcloud ones and Frankfurt (Germany, closed to Strasbourg) for AWS. The distance between the instance and the cluster are almost the same.

Time-to-First-Byte test

So what is the API efficiency on that new service? Here we want to compare the Time-to-First-Byte which give us an idea how the cluster manage to process the request, identify the object, check the right to access it and return it. So we’ll send tones of 1 Byte objects and measure the time needed to access it.

To be fare in that test, we need to take in consideration that we’ll get out to the OVHcloud network to reach the AWS cluster. Here are the ping we got :

OVHcloud instance to OVHcloud Object Storage

$ ping s3.sbg.perf.cloud.ovh.net
PING s3.sbg.perf.cloud.ovh.net (141.95.161.68) 56(84) bytes of data.
64 bytes from 141.95.161.68 (141.95.161.68): icmp_seq=1 ttl=53 time=10.2 ms

OVHcloud instance to AWS Object Storage

$ ping s3.eu-central-1.amazonaws.com
PING s3.eu-central-1.amazonaws.com (52.219.169.77) 56(84) bytes of data.
64 bytes from s3.eu-central-1.amazonaws.com (52.219.169.77): icmp_seq=1 ttl=45 time=11.0 ms

So we have almost the same latency, let’s go for the Time-to-First-Byte for each services.

OVHcloud High Perf looks a little bit better, not so far away but there is a small advantage for OVHcloud.

Small and medium files test

Now we’ll test not only the API but a normal object usage looking on files from 10KB to 10MB which represent plenty of usages, especially when it comes to serve statics on the web. That could represent code, images, sound…

As you can see, we leaved the field where the latency advantage where on OVHcloud side and we enter in the real performances using the whole chain including writing and reading on disks. Here, OVHcloud High Perf takes clearly the lead on 1MB files with around 55% faster than AWS S3 on download tests.

Large files test

When we move to 100MB files and more, we enter in a different space where the bandwidth is the main part which impact the global result. Here the data could be of course videos but also specific application files or large datasets.

Here, the upload performances are almost the same for AWS and OVHcloud but we still have a real difference on the download test with around 30% advantage in favor of OVHcloud High Perf.

Some take away about this new offer

Regarding the pure performances, OVHcloud High Perf Object Storage is a great challenger for AWS S3. The API is as fast as our competitor. We can notice a real advantage for OVHcloud on medium files (1MB to 100MB) where we go over 50% (1MB) faster than AWS on download and around 40% on upload (1MB).

Before discussing this in more detail we should understand why we benchmark in the first place. The number one reason is that we need to assess the impact on our customers before putting anything into production. This could be anything from a new storage platform, a new disk model, a kernel patch, to a firmware update and any number of use cases.

Short story

As we continue to improve our storage infrastructures and introduce new storage technologies we regularly enhance our benchmarking methodologies and tools. With thousands of hardware references and an equal number of configurations, the number of benchmarks we need to achieve is exponential. This is why it is vital to industrialise the process.

In an effort to increase transparency we chose to conduct our benchmarks from the point of view of the end user. This means that all the orchestration which we describe in this blog post can be done by any of our Hosted Private Cloud customers.

Tools & Orchestration

FIO, vdbench, I/O Meter, (dd !) are just some of the widely-used and proven tools to benchmark storage. They give you an overview of raw performance for a given storage type. But, what if you want to benchmark a whole infrastructure from end-to-end? This could include 1, 10, 100, 1000 VMs or more, with multiple disk/raid settings, multiple disk sizes, multiple block sizes. All with different generic workloads consisting of a certain percentage of reads/writes, sequential or random and run by so many threads. You would also need to use workloads which match your production workloads. In fact the combinations are endless.

With all this in mind, for our first iteration, we started to use HCIbench to automate our benchmarks.

Hyper-converged infrastructure benchmark

HCibench (https://flings.vmware.com/hcibench) is a free and open-source benchmark automation tool. It defines itself as a “Hyper-converged Infrastructure Benchmark”. It’s essentially an automation wrapper around the popular and proven open-source benchmark tools: Vdbench and Fio, making it easier to automate testing across an HCI cluster. HCIBench aims to simplify and accelerate customer POC performance testing in a consistent and controlled way. The tool fully automates the end-to-end process of deploying test VMs, coordinating workload runs, aggregating test results, performance analysis and collecting necessary data for troubleshooting purposes.

HCIbench is as easy to install as deploying an OVA (Open Virtual Appliance) in your VMware infrastructure:

Once HCIbench is set up, just point your browser to https://IP:8483 and you’re ready to start your benchmarks:

Once you have entered your vCenter credentials and some additional information (datacenter, cluster, network, datastore etc..),  you’ll be able to set guest VMs settings:

• Number of VMs to deploy
• Number of disks for each VM
• Disk size
• Benchmark tool (FIO or vdbench)
• I/O parameter file (see details below)
• Duration
• And more …

HCIbench will then take charge of all of the VM deployment/recycling operations and will execute your benchmark. The results are available in various forms, from Grafana interfaces to Excel files or simply flat text files for further external parsing…

Workload parameter files, modeling I/Os

Workload parameter files (using vdbench here) are at the heart of all benchmarks operations. They describe the I/O model you want to run against a given storage endpoint. Percentage of read/write , random/sequential, block size, threads and many other options are available.

We chose 3 different approaches to evaluate our storage platforms: generic workloads, application workloads and production workloads.

“Generic” workloads

By “generic” workloads we mean all workloads that look like “ONLY RANDOM READS” or “ONLY SEQUENTIAL WRITES”. They allow us to check how a storage type reacts on linear cases and how it performs with “extreme” cases.

Sample of a “generic” vdbench workload parameter file

root@photon-HCIBench [ /opt/automation/vdbench-param-files ]# cat 
GENERIC-ONLY-READS-RANDOM-16K-vdb-1vmdk-80ws-16k-100rdpct-100randompct-4threads
*SD:    Storage Definition
*WD:    Workload Definition
*RD:    Run Definitionsd=sd1,
        lun=/dev/sda,openflags=o_direct,
        hitarea=0,range=(0,80),
        threads=4,
        wd=wd1,
        sd=sd1,
        rd=run1,
        xfersize=16k,
        rdpct=100,
        seekpct=100,
        iorate=max,
        elapsed=120,
        interval=1

“Application” workloads

By “application” workloads we mean workloads that match common production use cases like “DATABASE WORKLOAD” , “VDI WORKLOAD” ,  “BACKUP WORKLOAD” etc.. With these benchmarks we can emulate a typical workload and verify the areas where a given storage type excels.

Sample of an “Application” vdbench workload parameter file

root@photon-HCIBench [ /opt/automation/vdbench-param-files ]# cat 
OLTP-SQL-Oracle-Exchange-vdb-1vmdk-80ws-16k-100random-70rdpct-4threads
*SD:    Storage Definition
*WD:    Workload Definition
*RD:    Run Definitionsd=sd1,
        lun=/dev/sda,
        openflags=o_direct,
        hitarea=0,range=(0,80),
        threads=4wd=wd1,
        sd=(sd1),
        xfersize=16k,
        rdpct=70,
        seekpct=100,
        rd=run1,
        wd=wd1,
        iorate=max,
        elapsed=120,
        interval=1

“Production” workloads.

Finally, another approach we are working on is the ability to “record” a production workload and “replay” it on an other storage endpoint to evaluate how the target storage performs with your production workload without the need to run your real production on it. The trick here is to use a mix of 3 tools blktrace , btrecord and btreplay to track and trace low level I/O calls and be able to replay those traces on another storage platform. 

In a future blog post we’ll share this feature with you, stay tuned!

Industrialisation of HCIbench runs with Rundeck scheduler

As we’ve seen, in few clicks we’re able to define and launch a benchmark scenario under a specific workload. Deployment and recycling of probing VMs are fully automated. What if, next, we want to iterate through multiple scenarios? As part of a whole validation of a new storage platform for example? At this point we started to use Rundeck (http://www.rundeck.com), a free and open-source runbook automation scheduler, in front of HCIbench. The idea is to be able to create complete benchmark scenario collections.

The first step was to understand how HCIbench works under the hood so we could control it via the Rundeck scheduler. HCIbench is designed to be used via a web interface but all the mechanics behind it are done via clean and separate scripts like start/stop/kill. All the benchmark settings are stored in a clean flat configuration file which was easy to convert as a template…

Templating HCIbench configuration file

root@photon-HCIBench [ /opt/automation/conf ]# cat template.yaml
vc: '<VCENTER_HOSTIP>'
vc_username: '<USERNAME>'
vc_password: '<PASSWORD>'
datacenter_name: '<DATACENTER>'
cluster_name: '<CLUSTER>'
number_vm: '<NUMBERVM>'
testing_duration: '<DURATION>'
datastore_name:- '<DATASTORE>'
output_path: '<OUTPUT_PATH>'
...

Bench “root job”

A rundeck job consists of a sequence of steps which will execute on a defined list of nodes. In our context nodes are VMs running HCIbench.

What we call the bench “root job” is a rundeck job which is the main bench entry point. Its generic role is to be called by other jobs and to launch one specific bench. 

Options (parameters) of this job are all the items from the HCIbench configuration template (see above).

The workflow for this job is as follows:

            – Parse job options
            – SSH connect to HCIbench VM
            – Populate configuration template with corresponding job options
            – Launch HCIbench 

Bench jobs

Secondly we have “bench jobs”. Through rundeck API we created jobs for each workload to be able to launch benches individually or by group, as required. Each of those “bench jobs” call the “root job” explained above with corresponding bench parameters.

“Super” jobs

Finally we have “Super jobs”. These are collections of jobs, their workflows are series of bench jobs calls. We use the cascading options mechanism to pass options through jobs. In the example below we bench a vSAN cluster through a complete panel of I/O models.

Another interesting feature of using Rundeck as an HCIbench scheduler is the ability to keep all logs from HCIbench’s VMs and the timing of each benchmark at the same place. Therefore, it’s easy to browse and search around all our benchmarks or to target a specific behaviour shown by graphics.

Results & use cases

vSAN

Integrating vSAN into our Hosted Private Cloud product was a typical benchmark project where we needed not only to check how the whole platform performed in every area but also to refine the design of the platform itself. On one side we evaluated hardware designs with multiple disk references and on the other side we improved the software design by evaluating various vSAN disk group and cache configurations.

New kernel impact on storage arrays

Another interesting use case is when we’re evaluating impacts of a new kernel on our OmniOS (http://www.omniosce.org) based storage arrays. OmniOS is a free open-source Operating System based on OpenSolaris which integrates some great technologies such as ZFS, DTrace, Crossbow, SMF, Bhyve, KVM and Linux zone support. This case shows not only slightly better performances but also a great improvement in I/Os handling.

Indeed, among a lot of different benchmarks, new kernel (r151022), shows far more stable and linear I/Os. This bench also confirms few ZFS/NFS fixes that were included in this kernel which fix latency issues during ZFS send/receive snapshot.

Industrialising our benchmarks enabled us to monitor the performance of our storage. First of all, because we created them with our users in mind, we’re aligned with what our customers would get. In addition, benchmarks give us an insight into troubleshooting storage issues that are very specific and/or only visible from VMs. We plan to extend this so we can check how the compute side performs (CPU/RAM/…). Finally, we’re now focusing on the record/replay workload feature allowing our users to predict how their production workload will perform on “XYZ” platforms without having to actually run their production environment on it. We will detail this in a future blog post, stay tuned!