Throughout the year, there are other ways we help the community with grants to access our infrastructure for testing or evaluation purposes. But that’s not all. Deploying at an unprecedented scale a number of open-source solutions we might find ourselves in the unique position to help the community through development and patching efforts. This recently happened within our storage team, and we thought sharing our experience would be worth a read.

OpenZFS and FreeBSD at OVHcloud

Among the Storage Product Unit, our mission is to deliver innovative file storage services based on different hardware and software stacks. We take the time to test and validate new stacks of technologies to deliver high performance and high availability storage with the utmost care for data protection all the while keeping costs reasonable. With a complete portfolio of storage solutions, we use different set of software foundations for different storage access modes including OpenZFS for file storage.

Based on our comprehensive testing, we chose FreeBSD for some of our offers like NAS-HA or Datastore NFS. One of the reasons is that FreeBSD is managed as a complete operating system with OpenZFS being a first-class citizen and natively integrated. It benefits from many years of experience across many teams ensuring quality and security.

FreeBSD’s release management goes through multiple steps from idea inception to public releases:

• Technical reviews by peers,
• Current branch to test,
• Stable branch for wider user base testing,
• Release candidate test,
• Normal release,

At the same time, patches for software are released to fix vulnerabilities and bugs.

The ports collection is well-designed and simple. While FreeBSD has binary packages, which are handled by the pkg package manager, it also has the ability to compile software from source, allowing user to select desired options of compilation.

FreeBSD also provides tools like poudriere. Poudriere is a utility for creating and testing FreeBSD packages. It makes it easy for users to build and set up their own binary package repository in which packages are built with their own options.

FreeBSD has over five hundred system variables that can be read and set using the sysctl utility. These system variables are used to apply some changes to a running FreeBSD system. This includes many advanced options of the TCP/IP stack and virtual memory system that can improve performance.

Our goal today is not to come up with an exhaustive list of technical advantages on why to use FreeBSD. It would probably require one to many full blog posts. Keep in mind jail, pf, linux binary compatibility and so on…

As we use FreeBSD, we are convinced that ZFS is a high-performance file system with replication, compression, encryption, and snapshots. If you want more details, our very own Frédéric Zind said it all during his tech talk at Very Tech Trip 2023 : 🇬🇧 / 🇫🇷

To illustrate what we are doing with OpenZFS and FreeBSD, let’s take an example with NAS-HA. This product is a file storage service (active/passive cluster illustrated below) built upon 2 nodes and a ZFS-based filesystem shared with NFS and/or CIFS. NAS-HA is a good example of how we integrated both FreeBSD and OpenZFS to build an open and secure storage service dedicated to versatile workloads (centralized storage for private or public cloud instances, bare metal servers…). With NAS-HA we are talking about several thousands of servers holding Petabytes of data.

Open-source contribution

To demonstrate how OVHcloud participates in open-source communities, we are sharing what we came to encounter in 2022 and how we fixed it.

Through our constant monitoring system, our operate team observed a condition where some filers had no more allocated memory due to a lack of free vnodes. As a reboot was necessary to unhang filers, the operate and engineering teams investigated the root cause. They found several leads but unfortunately couldn’t reproduce the incident. To improve the situation, the following was done:

• Double the base reservation of memory for OS (vfs.zfs.arc_max). Since our FreeBSD servers are booted into an in-memory OS without swap, we have to be careful that everything has sufficient resources and do not fight ZFS for memory,
• Create a buffer memory zone (vfs.zfs.arc.sys_free),
• Change the speed of release memory with no effect,
• Disable periodic checks because default checks were consuming vnodes (a vnode is a structure representing a filesystem entity like a file, directory, device node) and were unneeded for our use cases,

Even after applying these changes, some crashes were still happening. Then, we noticed that the incident seemed to be related to a non-holded snapshot during a restoration. There was a snapshot rotation while the snapshot was used during restoration, and it caused a deadlock. At this time, we still had no repro.

What we found:

• Huge spike in vnode usage,
• Spikes seemed to be correlated to the rotation and restoration of our backups (available through the .zfs directory on our NFS shares),

What we managed to:

• Limit the interaction of the .zfs directory to our essential operations,
• Manually set vnode limit to 10 million (via sysctl),

As a result, parallel rsync copies successfully finished. No more problems were observed during our quality assurance period (5 days), so we reached out to concerned communities.

Random servers were still experiencing incidents. The community started to report similar behaviors. In parallel, we noticed that if an NFS client browses the .zfs directory exposed by our servers all ZFS operations on snapshots would end up in a deadlock state afterwards. This happened only on OpenZFS 2.x (FreeBSD 13.x) and only a reboot would unfreeze the server. Problem reports have been escalated to OpenZFS (here) and Freebsd (here). FreeBSD’s users confirmed our testing with a procedure to reproduce the incident.

Thanks to OVHcloud’s FreeBSD developers, we were quickly pointed at a FreeBSD developer, Mark, in touch with ZFS matters and we let him know what we have done, found, and tested.

Leveraging the community’s feedback, Mark suggested patches and our engineering team tested each one of them. Communication was smooth and the root cause was finally identified: a Linux-specific race condition. Technically speaking the issue was described the following way: “zfsctl_snapdir_fid() sets fid_gen to 0 or 1 depending on whether the snapshot directory is mounted. On FreeBSD it fails, making snapshot dirs inaccessible via NFS.” (See commit on OpenZFS’s github)The patch was committed and merged in FreeBSD and OpenZFS’s repositories.

Why didn’t we catch this bug during our preprod tests? Because of mathematics. With fewer servers, there is lower probability to have concurrent access to .zfs directory. This test has since then been added to our test book.

Main takeaways

As a cloud actor, we are committed to open-source and many people at OVHcloud spend incredible amount of time to reproduce bugs, read code, test, and ultimately submit patches. Some of us even develop open-source projects (CDS, bastion by example are shining great examples) and/or during their free time.

At the Storage Product Unit, we are first and foremost geeks that love FreeBSD and OpenZFS. And well managing more than 10k servers daily is quite the experience.

Most importantly, we want to give back to open-source communities like Linux Foundation through the Ceph foundation, Cloud Native Computing Foundation, Python community, PostgreSQL community, FreeBSD, OpenZFS and more. If you want to learn more, you can check our open innovation program as well as OVHcloud GitHub repo.

Ces dernières années, l’épuisement des IPv4 est devenu une réalité, entraînant une augmentation des prix jusqu’à atteindre une moyenne de 50 dollars par IP en 2022. Malgré tout, nous avons continué à vous proposer nos services IPv4 au meilleur prix afin que vous puissiez vous concentrer sur la construction d’infrastructures adaptées et rentables. Cependant, nous devons maintenant ajuster nos prix pour refléter la situation actuelle du marché.

La transparence étant toujours au cœur de notre démarche, nous souhaitons vous communiquer les évolutions de prix suivantes.

À partir du 6 octobre 2022, le prix des nouveaux abonnements IPv4 supplémentaires (aussi connu sous le nom d’IP fail-over) sera de 1,50 € par mois et par IP (hors taxes), sans frais d’installation et sans engagement.

Pour les abonnements IPv4 additionnels existants et pour suivre le tarif des nouveaux abonnements nous intègrerons progressivement cette nouvelle tarification pour les Additional IP souscrites avant le 6 octobre 2022 :

1. À partir du 1er décembre 2022, les abonnements existants seront facturés au prix de 0,50 € par mois et par IP (hors taxes) ;
2. À partir du 1er avril 2023, les abonnements existants seront facturés au prix de 1,00 € par mois et par IP (hors taxes) ;
3. À partir du 1er septembre 2023, les abonnements existants seront facturés au prix de 1,50 € par mois et par IP (hors taxes).

Spécifiquement pour les offres So you Start 

Le tarif des abonnements (nouveaux et existants) au service Additional IP (IPv4 supplémentaires, géolocalisées et fail-over) évoluera progressivement comme suit : 

1. À partir du 1er mars 2023, les abonnements seront facturés au prix de 0,50 € HT par mois et par IP ; 
2. À partir du 30 mars 2023, au prix de 1,00 € HT par mois et par IP ; 
3. À partir du 1er septembre 2023, au prix de 1,50 € HT par mois et par IP. 

Notre engagement en matière d’innovation

Depuis 20 ans, OVHcloud a toujours livré des services complets et des produits innovants en investissant fortement dans son infrastructure et sa R&D. Grâce à cette approche, nous sommes en mesure de proposer de nouveaux services de connectivité : Bring Your Own IP, IP flottantes pour Public Cloud, Load Balancer as a Service, IPv6, etc.

Ces services peuvent vous aider à construire des architectures réseau fiables, réversibles et puissantes, pour soutenir votre propre innovation.

OVHcloud s’engage à continuer de servir sa communauté d’utilisatrices et utilisateurs en fournissant l’un des services IPv4 les plus compétitifs du marché.

• Nos adresses IPv4 supplémentaires offrent de multiples fonctionnalités : capacité de basculement, géolocalisation d’IP, reverse DNS.
• Garantir la réputation des adresses IP grâce à un investissement fort contre leur blacklisting.
• Dynamiser la productivité et l’expérience de gestion des adresses IP dans l’espace client OVHcloud.

Alternatives aux IPv4 supplémentaires

Comme alternative, si vous possédez vos propres adresses IPv4, nous avons récemment lancé le service Bring Your Own IP (BYOIP). Ce dernier vous permet de les importer vers OVHcloud afin de les utiliser comme adresses IPv4 supplémentaires grâce à la capacité de basculement. Pour en savoir plus sur le BYOIP, cliquez ici.

Toutes les adresses IPv4 supplémentaires (simples et blocs, importées ou non) peuvent être gérées depuis votre espace client OVHcloud, section « Network », puis « Adresses IP publiques » ou en utilisant les API OVHcloud dédiées.

Nous proposons également des adresses IPv6 sur la plupart de nos produits (serveurs Bare Metal, instances Public Cloud Compute, VPS, Load Balancer, etc.) et continuerons d’enrichir la liste des produits compatibles. Vous pouvez suivre les améliorations IPv6 à venir grâce à notre roadmap publique Infrastructure sur GitHub.

Notre équipe reste disponible pour discuter de l’état actuel du marché ou de votre situation et pour vous accompagner dans vos futurs projets. Pour toute question complémentaire, n’hésitez pas à nous contacter.

Quels produits seront concernés ?

Vous trouverez ci-dessous la liste des services concernés, ainsi que le nouveau tarif d’une adresse IPv4 et/ou d’un bloc d’IP supplémentaire :

Serveurs Bare Metal (dont serveurs Eco), projets Public Cloud et Load Balancer

VPS

Serveurs So you Start (Essential et Game)

vRack

Hosted Private Cloud powered by VMware

Over the past years, IPv4 exhaustion has become a reality. Prices continued to rise, reaching an average of $50 per IP in 2022. Through all of this, we’ve continued to provide IPv4 service at nearly no cost to you, so that you can focus on building appropriate and cost-effective infrastructures. However, now we need to adjust our pricing to reflect the current market situation.

As transparency is always key in our approach with our customers, we’d like to share the following price adjustments.

Starting on October 6^th, 2022, the price for new additional IPv4 subscriptions (also known as Failover IP) will be € 1.50 monthly per IP (exc. VAT), with no setup fee and no commitment.

For existing additional IPv4 subscriptions, and to follow the price of the new subscriptions, we will gradually integrate this new pricing for Additional IPs that have been subscribed to before October 6^th, 2022:

1. Starting on December 1^st, 2022, existing subscriptions will be charged at 0,50 € per month per IP (ex. VAT).
2. Starting on April 1^st, 2023, existing subscriptions will be charged at 1,00 € per month per IP (ex. VAT).
3. Starting on September 1^st, 2023, existing subscriptions will be charged at 1,50 € per month per IP (ex. VAT).

Specifically for So you Start offers 

The price of subscriptions (new & existing) for Additional IP services (additional IPv4, geolocated and fail-over) will gradually evolve as follow: 

1. From March 1, 2023, subscriptions will be charged at 0.50 € per month per IP (ex. VAT). 
2. From March 30, 2023, subscriptions will be charged at 1.00 € per month per IP (ex. VAT). 
3. From September 1, 2023, subscriptions will be charged at 1.50 € per month per IP (ex. VAT). 

Our innovation commitment

For the past 20 years, OVHcloud has always delivered end-to-end services and innovative products by strongly investing in our infrastructure and R&D. Thanks to this approach, we’re able to release brand new connectivity services: Bring Your Own IP service, Public Cloud Floating IPs, Load-Balancer-as-a-Service, IPv6 and more.

These services can help you build reliable, reversible, and powerful network architectures to support your own innovation.

OVHcloud is committed to continue serving our community of users by providing one of the most competitive IPv4 services on the market:

• Our additional IPv4 enable multiple features including failover capacity, IP geolocation, reverse DNS
• Ensuring quality IPs thanks to a strong investment in an unblack listing initiative.
• Boosting productivity with an upcoming refresh of the IP management experience in the OVHcloud Control Panel.

Additional IPv4 alternatives

As an alternative, if you have your own IPv4 ranges, we recently launched the Bring Your Own IP service allowing customers to import their own IPv4 addresses to OVHcloud, to use as additional IPv4 addresses with fail-over capability. Read more about the BYOIP here.

All additional IPv4s (single and blocks, imported or not) can be managed from your OVHcloud Control Panel, in the Network >> Public IP Addresses section or using OVHcloud APIs for IPs.

We also offer IPv6 on most of our products (Bare Metal servers, Public Cloud Compute instances, VPS, Load Balancer…) and will continue growing the IPv6-ready products list. You can follow upcoming IPv6 improvements on our Infrastructure public roadmap on GitHub: https://github.com/ovh/infrastructure-roadmap/.

Our teams remain available to discuss the current market situation, your specific situation, and support you in your future projects. For any additional questions, feel free to contact us.

What products will be impacted?

Below you’ll find the list of impacted services along with the new price for an additional single IPv4 and/or IP block:

Bare Metal servers (including Eco servers), Public Cloud projects and Load Balancer

VPS

SoYouStart servers (Essential and Game)

vRack

Hosted Private Cloud powered by VMware

And, while public sector data, financial records, customer information and confidential business reports were previously stored in secure facilities where physical access was restricted and closely monitored, in the digital age, vast amounts of data are continuously on the move.

These changes have seen the evolution of a security industry focused on ensuring confidential information is secure and can only be accessed by the people and systems authorised to do so.

Businesses are also realising and tapping into the value of the data they hold, with high-performance data processing driving whole new business models. With business data becoming more and more critical to operations, suddenly losing access to it would make it extremely difficult for many businesses to continue their activities.

The security challenges of high-performance workloads

High-performance computing is growing all the time, with the amount of data being extracted, transferred and stored increasing exponentially.

In addition to the sheer volume of data, high-performance workloads are complex, requiring a large number of nodes, distributed systems, and optimised configurations, which are tuned and tested for the specific business.

They must also deal with the added complexity that comes when data includes health records or financial information over different geographic regions, where local laws restrict the movement of certain types of information.

Add into the mix that these capabilities are increasingly provided in the cloud, and there are a number of extra considerations.

An expanded attack surface

With data being transferred to and from the cloud, as well as being processed and stored beyond the confines of organisation’s own IT infrastructure, there are more channels that malicious actors could use to intercept, disrupt or steal data.

Cloud providers’ data centres, which contain server hardware, network switches and IT tools, further expand this attack surface.

OVHcloud, for example, provides millions of public IP addresses across five continents and faces a constant battle to thwart attacks from hackers and criminal organisations hoping to profit from data theft, fraud and compromised systems.

This larger attack surface means cloud-based data processing is more attractive to malicious individuals or groups conducting distributed denial of service attacks (DDoS), where they attempt to overload an application or network with too much information.

High-performance workloads require high availability and lots of compute power by definition, so can be particularly impacted by these types of attacks. Not only can DDoS attacked overwhelm servers, they can also affect the general speed of traffic on the network — another issue that could particularly impact high-performance workloads.

Protecting high-performance workloads

Malicious actors present a constant threat to customer data, and so cloud providers of high-performance capabilities must go above and beyond when it comes to protecting the information on their networks.

They need to follow security best practice, use the most reliable and tested technology, and abide by regulations set out by governments and regulators.

Decentralising the computing and running high-performance workloads on numerous servers at the same time, means a successful attack on a server is less likely to affect the high-performance environment.

It’s also important for organisations to put disaster recovery and business continuity capabilities in place, so mission-critical workloads can continue to safely and securely operate even if a security issue occurs.

Being aware of the combined security implications of high-performance workloads and the cloud, means organisations can plan the right approach and select the cloud provider that will best manage their risks.

AI, for example, is being used in applications that recognise human emotions or mimic human conversation. Some multinationals, meanwhile, are developing applications to automate back-office processes, power intelligent chatbots for customer service, or that can scan millions of information-heavy documents to find insights.

What unites these technologies is that they involve large amounts of data that must be rapidly and reliably processed. This has created an increased need for businesses to have access to high-performance workload capabilities.

One piece of research suggests the high-performance computing market will grow at a compound annual growth rate of 5.5 per cent between 2020 and 2025 to reach $49.4 billion.

However, many organisations struggle to overcome resourcing issues that can either prevent these capabilities being put in place or stop companies from exploiting them to their full potential.

The resource question

One issue is around cost. The hardware, software and networks aren’t the standard kit being used for other business operations, but specialist products that come with a premium price tag.

In addition, many organisations lack the time to get these capabilities up and running. This is particularly true when you consider the additional burden placed on IT departments due to the Covid-19 pandemic, during which they’ve had to support an increase in remote working and manage the increased risk associated with shadow IT.

High-performance workloads are complex, requiring a large number of nodes, distributed systems and data. While this may all sound quite daunting, it doesn’t have to be — particularly with the right approach and service provider.

To gain high-performance capabilities, businesses don’t have to go all-in — they can start small, both in terms of cost and capabilities, before expanding their usage.

How cloud helps out

One major reason for this is the fact that high-performance capabilities can now be accessed via the cloud. There is no need to invest in expensive and complicated on-premises equipment — this will all be provided by the service provider. No additional space needs to be found and there is no need to have IT staff who are dedicated to the upkeep of the infrastructure.

Service providers will also be able to provide the networking capabilities and tools needed to keep the high-performance capabilities — which are often business critical — available at all times and can support large amounts of data securely.

Even better, businesses only need to pay for the capabilities they use — there is no need to pay for unused capacity on the off chance it could be needed in the future. A business can experiment with small implementations before deciding whether they want to up their capabilities, meaning little money is wasted and there is minimal risk.

A service provider with the answers

In addition to the benefits that cloud offers, service providers can also address other resource challenges around high-performance workloads.

When it comes to pricing, transparency is key. It’s a particularly big advantage for companies that have limited budgets as they are able to have predictable costs, even if they have an unexpected spike in traffic or need to restore a large backup.

Another advantage is having the ability to take on dedicated high-performance servers individually, which they can build on to eventually create a cluster, or clusters. This flexibility to start small really helps businesses dip their toe in the water with high-performance workloads, before embarking on anything more ambitious.

This makes it easy for organisations to gain the capacity they need — so business requirements are fully met without resources being wasted or overcommitted. OVHcloud, for example, offers several storage configuration options so organisations can choose the right disk configuration based on their needs.

Conclusion

With the ability of cloud technology to support data-intensive workloads, and service providers that address the other resource issues, there is no need for businesses to think high-performance capabilities aren’t a realistic option.

They can start playing with small stakes before going big — all with minimal risk. If the new capabilities fulfil the business need, they can be retained or scaled up. And if capabilities fail to meet business needs, businesses can stop using the services without significant financial cost.

In this way, your business can enjoy the benefits of high-performance workloads without putting a strain on resources, opening up a whole new world of possibilities.

According to IDC, more than 55 billion devices, including industrial IoT applications and sensors, will generate more than 73ZB of data by 2025, up from 18ZB in 2019. The analyst firm also expects worldwide storage capacity to increase from 6.8ZB in 2020 to 8.9ZB by 2024.

With the volume of data growing exponentially and technologies improving quickly to handle that growth, the latest and most cutting-edge computing hardware doesn’t come cheap. As a result, organisations need to evolve their procurement practices to better serve customers, improve their bottom line and remain competitive.

Traditionally, when an organisation is planning an update to its IT infrastructure, it sets a budget and finds the solution to fulfil its technical requirements at the right price. Technology upgrades, meanwhile, are often seen as an opportunity to cut costs rather than enhance profitability.

By focusing on the pure dollar amount of a project, organisations are missing the potential competitive advantage that could be gained by employing high-performance capabilities in to rapidly transform raw data into business insights.

Can you afford to?

Cloud technology is well established but high-performance capabilities add a whole new dimension by creating a wave of opportunity for innovation, especially around workloads that contain enormous amounts of data and employ machine learning, pattern recognition and AI.

High-performance systems are built from components that are orders of magnitude faster than standard technologies and can provide a level of performance that until recently was only available to large institutions and government departments with deep pockets.

However, over the past few years, advances in technology have resulted in high-performance capabilities becoming more accessible than ever before, creating an opportunity for organisations that are able to break from the traditional IT procurement mindset.

Can you afford not to?

Instead of looking at IT plans and considering the technologies required to keep doing the same thing, innovative organisations are imagining how much more competitive, efficient and profitable they could be if they were able to run high-performance workloads to create a performance gap between themselves and the competition.

High-performance products sell at a premium compared to standard cloud infrastructures but the additional costs can be recouped with improved efficiencies, scalability and competitiveness.

The next wave of disruptors will use high-performance capabilities to rapidly process, organise and analyse their data to create innovative products, fight off competition and create completely new markets.

According to a recent study by Mordor Intelligence the value of the global high-performance computing industry will increase from $24 billion in 2020 to over $42 billion by 2026.

High-performance systems are quickly becoming the standard for big data workloads, meaning that any organisation wanting to remain competitive in this area will have to determine how high-performance workloads can be integrated into its infrastructure.

Why high-performance capabilities make sense

Choosing to implement high-performance capabilities is a complex task that requires technical expertise and a solid business plan.

Organisations need to identify the most suitable processor, bandwidth and memory for their workloads. Do they have the right security, will their systems be interoperable, and do they really know exactly how much the whole thing will cost?

Some companies may find it counterintuitive to be spending vast sums of money on cloud-based high-performance capabilities when a regular cloud infrastructure could do a similar job but slower. If their workloads are not time critical then maybe spending on premium technology can’t and shouldn’t be justified.  But if workloads are time critical, if you deal in real time data, if AI and machine-learning systems are required to analyse and manipulate data to generate critical information, a change of mindset is all you need to access the high-performance capabilities your business needs.