Our customers have been able to build, deploy and run their applications in the cloud by leveraging our Public Cloud, a coherent and integrated ecosystem of infrastructure and managed services. As our customers’ cloud applications evolve, the need for ready-to-use, managed services has become more prevalent, as it enables them to focus on adding more capabilities to their apps. While customers have outsourced to cloud providers the responsibility of managing foundational infrastructure services, they are looking for services that make it easier for them to concentrate on their business logic and network services are no exception to this trend. In this post, we’ll highlight the new services we are building to meet the needs of these types of customers.

Network managed services

Though the network is thought of as an infrastructure component, it is interesting to differentiate the raw capability of providing network connectivity from the network-related services like routing, load-balancing, and traffic management. In the past, it was possible to add these capabilities on your own using compute instances and deploying software on top of the components that perform the aforementioned network-related services. The main benefit of the services discussed in this post is that customers no longer need to implement and maintain such services, as we now provide them.

Load Balancer

While our Kubernetes customers have been able to use our Load Balancer, integrated with our Managed Kubernetes, we now provide a network and application-level load balancer compatible with our Public Cloud Openstack deployment. This load balancer is built upon the OpenStack Foundation’s Octavia project and can be managed through Openstack tools (API, CLI, or Horizon). This is a great fit for users that are familiar with the OpenStack environment and are looking for an infra-as-a-code approach. Load Balancer is available in a variety of sizes to accommodate different usage and performance levels and provides typical cloud features: dynamic public or private environments, high availability, and an interface to cloud-native platforms. Check out all the details on our product page: https://www.ovhcloud.com/en-ie/public-cloud/load-balancer/.

Gateway

Enabling to connect networks, Gateway brings more flexibility to manage resource access through the network. A typical Gateway use case is to split your architecture on multiple networks and configure routes on demand. Gateway can also be an internet gateway, with instances using private ports to remain isolated from the public. In addition to classic private network management tools, Gateway brings more granular management to your networks. More details here: https://www.ovhcloud.com/en-ie/public-cloud/gateway/

Floating IPs

Floating IPs are public IPs that can move from one instance to another and are handled by Gateway to forward traffic to the assigned public IP. There are several situations where floating IPs are useful:

# You can define a floating IP for your application in advance, without having to create an instance to get a public IP. This can help you manage DNS records.
# You can link your floating IP to instances reachable through your virtual router, allowing you to enable internet ingress traffic on-demand.
# You can switch the link from one instance to another at any time, which can help you manage outage situations, or simply schedule maintenance actions (running rolling upgrades on multiple nodes, for example). Find out more about the product here: https://www.ovhcloud.com/en-ie/public-cloud/floating-ip/.

What’s in for me?

Here, we’ll take some concrete examples so that you can better understand what each service can do for you. Floating IP, Gateway, and Load Balancer can be deployed together to create the appropriate rules for network accessibility while providing the required security. The following scenarios highlight the key benefits of each service used in combination.

Floating IP scenarios

Gateway scenarios

Load Balancer scenario

Next steps

A big congrats to our teams that made the release of our Load Balancer, Gateway, and Floating IPs services available to our customers! At launch our network services, are available in the following Public Cloud locations: GRA9, GRA5, GRA7, SBG5, SBG7, UK1, DE1, WAW1, BHS5, SYD1, SGP1. After the launch, keep track as additional locations are rolled out here.

For new customers to our Public Cloud, you can take advantage of a free trial and test these new services along with our full catalog. We are always looking for feedback, feel free to reach out on our Discord channels. Last but not least, check out our publicly available roadmap on Github and propose new features and interact with our Product Managers.

A snapshot is a mechanism that allows you to create a new image from a running instance. This mainly serves two purposes:

1. As a backup mechanism: save the main disk of your instance to an image and later boot a new instance from this image with the saved data.
2. As a templating mechanism: customise a base image and save it to use as a template for new instances.

Snapshots can be taken of instances while they are either running or stopped. In simple terms, snapshots are images with the following additional properties:

How to create a snapshot

Using the CLI

To create a snapshot of an instance using the CLI, use the following command:

# Load your OpenStack credentials
$ source openrc

# Using the openstack client
$ openstack server image create --name <name of the new image> <instance name or uuid>

# Or using the nova client (deprecated)
$ nova image-create <instance name or uuid> <name of the new image>

Using Horizon

Once you’re logged in to Horizon, you can create a snapshot via the Compute → Instances page by clicking on the “Create snapshot” action.

The snapshot’s status and information about it can be found on the Compute → Images page.

You can then select the snapshot when creating a new instance.

Live snapshots and data consistency

We call a snapshot taken against a running instance with no downtime a “live snapshot”.

These snapshots are simply disk-only snapshots, and may be inconsistent if the instance’s OS is not aware of the snapshot being taken.

This phenomenon occurs when a hypervisor freezes the instance to allow the creation of a “delta” file before resuming the execution of the instance. This is done to prevent the instance writing directly to its disk while it is copied. When the copy is done, the instance is frozen again to allow the “delta” to be merged with the instance’s disk, and the execution is then resumed with the disk fully merged.

Inconsistencies can appear on the first freeze if the instance is not aware that the hypervisor is taking a snapshot, because the applications and the kernel running on the instance are not told to flush their buffers.

Ensuring snapshots are consistent

OpenStack Nova relies on QEMU to manage the virtual machines. QEMU also provides tools to communicate with an agent installed on the instance, in order to allow it to take certain actions before a snapshot.

This communication takes place via a virtual device added to the instance when OpenStack Nova detects that the image used has the following property: hw_qemu_guest_agent set to yes.

On a previously created private image, you can set the properties using this command:

# Using the openstack client
$ openstack image set --property hw_qemu_guest_agent=yes <image name or uuid>

# Or using the glance client (deprecated)
$ glance image-update --property hw_qemu_guest_agent=yes <image name or uuid>

To check the properties are indeed set, use the following command:

$ openstack image show -f value -c properties <image name or uuid>

The following diagram shows the workflow of a snapshot under these conditions:

The specific step that prevents any inconsistencies is #6: QEMU-agent freezes the file filesystem.

Configuring the QEMU agent

Linux

The qemu-guest-agent is not installed by default, but once it is installed and started, the filesystem freeze/thaw mechanism will work straight out of the box.

You can check that your instance is set up for communication with the hypervisor by checking the specific device:

$ file /dev/virtio-ports/org.qemu.guest_agent.0
/dev/virtio-ports/org.qemu.guest_agent.0: symbolic link to `../vport2p1'

If this file is not present, the qemu guest agent will not work, which means your image does not have the hw_qemu_guest_agent property set to yes.

Debian-based distributions (Debian, Ubuntu)

# Install the agent
user@agent:~$ sudo apt-get update
user@agent:~$ sudo apt-get install qemu-guest-agent

# Check the agent is started (it should be automatically started and enabled)
user@agent:~$ sudo service qemu-guest-agent status

Redhat-based distributions (Centos, Fedora)

# Install the agent
user@agent:~$ sudo yum install qemu-guest-agent

# Enable the agent
user@agent:~$ sudo chkconfig qemu-guest-agent on

# Start the agent
user@agent:~$ sudo service qemu-guest-agent start

# Check the agent is started
user@agent:~$ sudo service qemu-guest-agent status

Windows

Download and install the MSI related to your architecture (32 or 64 bit versions, although we recommend 64 bits for Public Cloud) from the Fedora project: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-qemu-ga/

You can check the service is running by using this powershell command:

PS C:\Users\Administrator> Get-Service QEMU-GA

Status   Name               DisplayName
------   ----               -----------
Running  QEMU-GA            QEMU Guest Agent

The Fedora documentation on creating Windows images with virtIO drivers can be found here. [2]

Advanced usage: QEMU agent hooks

It is possible to add scripts that will run before the filesystem is frozen by the agent and after it is thawed.

The example below is done on Debian 9, so the configuration might need to be adjusted for a different distributions.

Also, some distributions already provide the fsfreeze-hook.

Add the fsfreeze-hook script

First, we need to add and activate the fsfreeze-hook mechanism:

# Create the folders to receive the hooks
debian@agent:~$ sudo mkdir -p /etc/qemu/fsfreeze-hook.d

# Download the fsfreeze-hook from the QEMU repository
debian@agent:~$ sudo wget -O /etc/qemu/fsfreeze-hook https://raw.githubusercontent.com/qemu/qemu/master/scripts/qemu-guest-agent/fsfreeze-hook
debian@agent:~$ sudo chmod +x /etc/qemu/fsfreeze-hook

# Add the configuration of the qemu-guest-agent daemon to use this script
debian@agent:~$ sudo tee /etc/default/qemu-guest-agent > /dev/null <<EOF
DAEMON_ARGS="-F/etc/qemu/fsfreeze-hook"
EOF

# Restart the service to take the modifications into account
debian@agent:~$ sudo service qemu-guest-agent restart

Example hook script

The /etc/qemu/fsfreeze-hook script allows users to add custom scripts, to be run before and after the filesystem freeze.

Let’s add a test that writes in a file when the instance is being frozen and thawed:

debian@agent:~$ sudo tee /etc/qemu/fsfreeze-hook.d/test_hook.sh > /dev/null <<EOF
#!/bin/bash

case \$1 in
 freeze)
   echo "I'm frozen" > /tmp/freeze
   ;;
 thaw)
   echo "I'm thawed" >> /tmp/freeze
   ;;
 *)
   exit 1
   ;;
esac
EOF

debian@agent:~$ sudo chmod +x /etc/qemu/fsfreeze-hook.d/test_hook.sh

Be very careful with custom hook scripts. If one fails, the snapshot will be totally abandoned and destroyed. If you cannot find the image that was supposed to be created with the snapshot, it is probably because one of the scripts has failed. In this case, check the instance’s qemu-agent log.

Take a snapshot of your instance:

$ openstack server image create --name test_snapshot <instance name or uuid>

Check the test hook has been run:

# It works!
debian@agent:~$ sudo cat /tmp/freeze
I'm frozen
I'm thawed

Clean the test:

debian@agent:~$ sudo rm /etc/qemu/fsfreeze-hook.d/test_hook.sh /tmp/freeze

Why isn’t this enabled by default?

The qemu-guest-agent is not installed by default on most distributions. So why don’t we add it by default on the base images we provide?

The problem is that we are not legally allowed to change the content of the base images provided by Ubuntu, so for the sake of uniformity, we don’t install the package on any distribution.

Sources:

• Sébastien Han’s “OpenStack: Perform Consistent Snapshots” blog entry [1]
• The proxmox wiki article on QEMU guest agent [2]
• The Fedora documentation on creating Windows images with virtIO drivers [3]
  
  

Files inside files

We settled on a simple approach: we would store all these small fragments in larger files. This means that inodes usage on the filesystem is much lower.

These large files, which we call “volumes”, have three important characteristics:

• They are dedicated to a Swift partition
• They are append only: never overwrite data
• No concurrent writes: we may have multiple volumes per partition

We need to keep track of the location of these fragments within a volume. For this we developed a new component: the index-server. This will store each fragment location: the volume it is stored in, and its offset within the volume.

There is one index-server per disk drive. This means that its failure domain matches with the data it is indexing. It communicates with the existing object-server process through a local UNIX socket.

Leveraging on LevelDB

We chose LevelDB to store the fragment location in the index-server:

• It sorts data on-disk, which means it is efficient on regular spinning disks
• It is space efficient thanks to the Snappy compression library

Our initial tests were promising: it showed that we need about 40 bytes to track a fragment, vs 300 bytes if we used the regular file-based storage backend. We only track the fragment location, while the filesystem stores a lot of information that we don’t need (user, group, permissions..) This means the key-value would be small enough to be cached in memory, and listing files would not require physical disk reads.

When writing an object, the regular swift backend would create a file to hold the data. With LOSF, it would instead:

• Obtain a filesystem lock on a volume
• Append the object data at the end of the volume, and call fdatasync()
• Register the object location in the index-server (volume number, and offset within volume)

To read back an object:

• Query the index-server to get its location: volume number and offset
• Open the volume and seek to the offset to serve the data

However, we still have a couple of problems to solve!

Deleting objects

When a customer deletes an object, how can we actually delete data from the volumes? Remember that we only append data to a volume, so we can’t just mark space as unused within a volume, and try to reuse it later. We use XFS, and it offers an interesting solution: The ability to “punch a hole” in a file.

The logical size does not change, which means that fragments located after the hole do not change offset. The physical space, however, is released to the filesystem. This is a great solution, as it means we can keep appending to volumes, free space within a volume, and let the filesystem deal with space allocation.

Directory structure

The index-server will store object names in a flat namespace, but Swift relies on a directory hierarchy.

/mnt/objects/<partition>/<suffix>/<checksum>/<timestamp>.data

The partition directory is the partition which the object belongs to, and the suffix directory is just the last three letters of the md5 checksum. (This was done to avoid having too many entries in a single directory)

If you have not used Swift before, the “partition index” of an object indicates which device in the cluster should store the object. The partition index is calculated by taking a few bits from the object’s path MD5. You can find out more here.

We do not explicitely store these directories in the index-server, as they can be computed from the object-hash. Remember that the object names are stored in order by LevelDB.

Data migration

This new approach changes the on-disk format. However we already had over 50 PB of data. Migrating offline was impossible. We wrote an intermediate, hybrid version of the system. It would always write new data using the new disk layout, but for reads, it would first look up in the index-server, and if the fragment wasn’t found, it would look up the fragment in the regular backend directory.

Meanwhile, a background tool would run to slowly to convert data from the old system to the new one. This took a few months to run over all machines.

Results

After the migration completed, the disk activity on the cluster was much lower: we observed that the index-server data would fit in memory, so listing objects in the cluster, or getting an object’s location would not require physical disk IO. The latency improved for both PUT and GET operations, and the cluster “reconstructor” tasks were able to progress much faster. (The reconstructor is the process that rebuilds data after a disk fails in the cluster)

Future work

In the context of object storage, hard drives still have a price advantage over solid state drives. Their capacity continues to grow, however the performance per drive has not improved. For the same usable space, if you switch from 6TB to 12TB drives, you’ve effectively halved your performance.

As we plan for the next generation of Swift clusters, we must find new ways to use these larger drives while ensuring performance is still good. This will probably mean using a mix of SSDs and spinning disks. Exciting work is happening in this area, as we will store more data on dedicated fast devices to further optimise Swift’s cluster response time.

We are using it at OVHcloud for internal needs, and as a service for our customers.

By design, it is rather easy to use, but you still need to think about your workload when designing a Swift cluster. In this post I’ll explain how data is stored in a Swift cluster, and why small objects are a concern.

How does Swift store files?

The nodes responsible for storing data in a Swift cluster are the “object servers”. To select the object servers that will hold a specific object, Swift relies on consistent hashing:

In practice, when an object is uploaded, a MD5 checksum will be computed, based on the object name. A number of bits will be extracted from that checksum, which will give us the “partition” number. 

The partition number enables you to look at the “ring”, to see which server and disk should store that particular object. The “ring” is a mapping between a partition number, and the object servers that should store objects belonging to that partition.

Let’s take a look at an example. In this case we will use only 2 bits off the md5 checksum (far too low but much easier to draw! There are only 4 partitions)

When a file is uploaded, from its name and other elements, we get a md5 checksum, 72acded3acd45e4c8b6ed680854b8ab1. If we take the 2 most significant bits, we get partition 1. 

From the object ring, we get the list of servers that should store copies of the object. 

With a recommended Swift setup, you would store three identical copies of the object. For a single upload, we create three actual files, on three different servers.

Swift policies

We’ve just seen how the most common Swift policy is to store identical copies of an object.

That may be a little costly for some use cases, and Swift also supports “erasure coding” policies.

Let’s compare them now.

The “replica policy” which we just described. You can choose how many copies of objects you want to store.

The “erasure coding” policy type

The object is split into fragments, with added redundant pieces to enable object reconstruction, if a disk containing a fragment fails.

At OVHcloud, we use a 12+3 policies (12 pieces from the object and 3 computed pieces)

This mode is more space efficient than replication, but it also creates more files on the infrastructure. In our configuration, we would create 15 files on the infrastructure, vs 3 files with a standard “replication” policy.

Why is this a problem?

On clusters where we have a combination of both an erasure coding policy, and a median object size of 32k, we would end up with over 70 million files *per drive*.

On a server with 36 disks, that’s 2.5 billion files.

The Swift cluster needs to regularly list these files to:

• Serve the object to customers
• Detect bit rot
• Reconstruct an object if a fragment has been lost because of a disk failure

Usually, listing files on a hard drive is pretty quick, thank’s to Linux’s directory cache. However, on some clusters we noticed the time to list files was increasing, and a lot of the hard drive’s IO capacity was used to read directory contents: there were too many files, and the system was unable to cache the directory contents. Wasting a lot of IO for this meant that the cluster response time was getting slower, and reconstruction tasks (rebuilding fragments lost because of disk failures) were lagging.

In the next post we’ll see how we addressed this.

Together, we explored the topic of continuous deployment in the context of machine learning and big data. We also discussed continuous deployment for environments like Kubernetes, Docker, OpenStack and VMware VSphere.

If you missed it, or would like to review everything that was discussed, you can listen to it again here. We hope to return soon, to continue sharing our passion for testing, integration and continuous deployment.

Although the podcast was recorded in French, starting from tomorrow, we’ll be delving further into the key points of our discussion in a series of articles on this blog.

Find CDS on GitHub:

• https://github.com/ovh/cds

…. and follow us on Twitter:

• https://twitter.com/yesnault
• https://twitter.com/francoissamin

Come chat about these subjects with us on our Gitter channel: https://gitter.im/ovh-cds/