Ever since 2003, OVHcloud has developed watercooling at scale for its servers in a free cooling approach that prohibits the use of power-hungry air conditioning thus reducing power costs and water usage. Over the years, our technology has been refined from generation to generation. All these years of work allowed us to see that one crucial but often overlooked parameter is the Delta T, a parameter that measures the difference in temperature for the water coming in and out of the data centre. From this single parameter comes greater efficiency particularly when it comes to Water Usage Effectiveness (WUE). A parameter that we continue refining with the most recent milestone being the Delta T=20K. Ready for the deep dive?

Overview of OVHcloud cooling DC principles

A typical OVHcloud data centre benefits from an indirect free cooling technique, also known as IFC, with the deployment at scale of a liquid cooling system. It helps in improving global data centre power efficiency with up to 50% energy savings compared to conventional mechanical cooling. With such a cooling strategy that leverages liquid cooling and air cooling, OVHcloud’s data centres aim at being highly efficient.

Our proprietary watercooling system recovers fatal heat generated by our servers through carefully crafted waterblocks offering direct contact with the chips. If we break down the system, there are four cooling bricks in the data centre:

• Dry Cooler (heat rejection system),
• Pumping Substation (PSS),
• Rack Rear Door Heat Exchanger (RDHX) and Cooling Distribution Unit (CDU),
• Server waterblock,

A typical water circuit in an OVHcloud data center goes like this: cold water is pumped to the rack from a pumping substation (PSS) before reaching the Rear Door Heat Exchanger (RDHX) installed at the rack level. Each rack comes with 3 RDHXs, each door having its own set of fans supporting air cooling capacity. Water then flows through the CDU or Cooling Distribution Units: placed inside the rack, CDU are comprised of pumps and plate heat exchangers (PHEXs). They distribute liquid to the waterblocks, effectively acting as a watercooling system, where the IT equipment heat will be transported back to the dry coolers located outside of the data centre for heat rejection.

Playing an important role in the cooling infrastructure, the dry coolers use external air to cool down the circulating liquid pumped from the data centre without any refrigeration. Depending on the climatic conditions, the most efficient evaporative cooling (EC) system to date is used to cool the outside air before it reaches the fins of the dry cooler.

Kelvin, how delta T is measured and why it’s a key parameter

In the research field, the usual norm is to express temperature difference in Kelvin while all temperature measurements are in °C, F or K. In this document, temperature is express in °C, while temperature difference is expressed in K. When it comes to our data centres, Delta T is the difference of water temperature between the inlet and outlet of a rack (in other words the in/out water temperature of the data center): the greater this difference is, the more efficient the data centre will be. Why? An increase in the temperature difference in the racks (dT) causes a decrease in the flow rate of water supplied per rack allowing the use of lower pumps capacity. Thus, an increase in the temperature difference permits a higher heat load per pumping unit and dry cooler for an equivalent PSS hydraulic capacity and heat rejection system.

From Delta T=5K to Delta T=20K

As we have seen, from a greater temperature difference comes more efficiency. It also means that the performance of the dry cooler is significantly improved. Indeed, water enters at higher temperature levels, while fan and evaporative cooling water consumption per kWh IT is reduced because a lower air flow rate is required. That later consequence allows the optimization of the ventilation capacity.

Since 2019, we upgraded to a new generation of cooling infrastructure for our most recent data centres capable of operating at a Delta T=20K instead of Delta T=5K. We rely on a lower number of dry coolers per data centre, yet more efficient ones, with smaller pumps and reduced piping. The result is a potential higher footprint (kW/m2) for the data centre, lower overall carbon footprint, better PPUE and an improved WUE, as well as reduced CAPEX and OPEX.

• Delta T=5K explained

In a traditional OVHcloud data centre with a Delta T=5K setting, water temperature is to be maintained at 25°C providing inlet air temperature at 30°C and inlet water temperature of the waterblock at 35°C. As shown in the figure below, two circuits in parallel are connected to the rack.

Through constant development, we improved the cooling design of our rear door heat exchanger (RDHX) to increase the inlet water temperature and reduce pinch between cold air and cold water: this way, the data center inlet water temperature is now set at 27°C against 25°C previously.

• Delta T=20K explained

Building up on the notable improvements we discussed, we were also able to setup a serial circuitry with PHEX and RDHX on the rack level in order to achieve a complete Delta T=20K environment as shown in the figure.

With water temperature maintained at 27°C, the inlet air temperature of the data centre remains at 30°C and inlet water temperature at 45°C for the waterblock. The data centre water outlet temperature is set at 47°C. On a sidenote, our robust design gives us confidence in the ability to operate with a higher temperature profile, still in a Delta T=20K setting, like 32-52°C.

What are the benefits of Delta T=20K VS Delta T=5K? And what are the advantages when applied to different data centre locations?

To measure the gains of OVHcloud’s IFC system operating with a Delta T=20K, we conducted an extensive series of benchmarks in various locations with the same Cooling Module configured first as Delta T=5K then as Delta T=20K. We should be able to highlight significant gains in terms of water consumption.

We elected for the following methodology. We collected and analyzed the weather data for the past 40 years, including temperature and relative humidity with a record every 30 minutes. For the Roubaix site for example, the warmest year happened to be 2019. Figure below shows the temperature frequency in hours extracted from climatic data recorded at Roubaix in 2019. The key hours are between 5 and 20 °C.

Based on our working principle, we can deduct the WUE in every country where we have a data centre. For your reference, in this scientific paper we estimated the theoretical WUE for some of our locations that offer the best possible representation of various climatic conditions: Roubaix (France), Mumbai (India), Singapore, VintHill (USA). For accuracy purposes we’re sharing below the estimated WUE for Roubaix:

This theoretical result is impressive. Yet, it’s important to have a better sense of what WUE can represent: we estimate that in an IFC with a Delta T=20K setting, 381 m³ of water are enough to cool a 600 kW DC versus 964 m³ in an IFC with Delta T=5K setting.

The table below is a summary of the most glaring changes between Delta T=5K and Delta T=20K:

Versus the competition?

To date, there are two different families of water liquid cooling techniques employed within data centres: close-coupled cooling and direct liquid cooling (see table). Watercooling solutions take advantage of the heat-carrying capacity of the water that is 3,3 times higher than that of air for a dramatic reduction in thermal resistance. With CPU and GPU being the most power-hungry parts in a server, liquid cooling via direct-to-chip contact contributes in returning warmer water from IT equipment than that of closely coupled cooling systems. Logically, the coolant temperature directly impacts energy savings and cooling efficiency participating in a significant decrease in the total DC site energy consumption by 14 to 20%¹. Please note, more information on the above can be obtained here.

What’s next?

As we celebrate the 20th anniversary of our watercooling technology this year, we continue to innovate. We have been working on designing a new breed of data centre employing climate friendly technologies when it comes to cooling. In a global context of surging energy prices and several water shortages in the 2022 summer, it becomes quite obvious as to why our previous technological bets pay off. With best-in-class PUE and WUE for our data centres, we continue to work tirelessly on improving our footprint within the ecosystem illustrating our deep commitment towards sustainability. Hopefully by now you have come to realize how important to the data centre is the Delta T parameter. From the infrastructure point of view to the tangible water efficiency, Delta T=20K contributes in adapting to various geographies and ultimately water conservation.

The water consumption of data centres has indeed a tremendous impact locally going that far as to affect river flow, groundwater levels and influence overall water resources for a whole territory. Thus, reducing WUE proves critical for any modern data centres to lower the overall impact on local communities.

The impressive saving of water illustrated by our WUE also contributes to a streamlined PUE for compliance with ISO/IEC 30134 standard. Last, but not least, the demonstrated advantages of Delta T=20K will prove to be a solid basis for the next generation of improvements our teams develop.

1: Performance analysis of new liquid cooling topology and its impact on data centres –  https://www.sciencedirect.com/science/article/abs/pii/S1359431122006779?via%3Dihub

Here at OVHcloud, we pioneered free cooling, and have used water cooling at scale to cool down our servers since 2003. This unique approach helped us to set high standards in PUE/WUE indexes for our global datacentres, all the while lowering our overall carbon impact.

With the evolution of IT equipment power, OVHcloud R&D worked on a new generation of datacentre cooling solutions. One of the most promising technologies today is immersion cooling, thanks to direct contact of the fluid with hardware components. It allows us to build datacentres all over the world maintaining low PUE/WUE indexes even in harsh climatic zones.

In this blog post, we will explain how we developed our own breed of immersion cooling to achieve several goals, from improving datacentre footprint and reducing power consumption, to adapting to new operating conditions and improved overall reliability.

OVHcloud Hybrid Immersion Liquid Cooling

Immersion Cooling is the practice of submerging electronic devices in a thermally and not electrically conductive liquid.

The typical immersion cooling solution usually employs pumps, heat sink structures, heat exchangers, condensers, sealed evaporative equipment etc. that either consume large amounts of energy to operate, require sealed casings, or occupy relatively large surface areas that limit the number of servers that can be implemented.

OVHcloud developed its own Hybrid Immersion Liquid Cooling technique. It is comprised of a direct to chip water cooling system and a passive natural single phase immersion cooling system containing two fluids:

• Water: cooling a heat sink through water blocks disposed on CPUs and GPUs with the same solution used in all OVHcloud servers and a proprietary serpentine convection coil connected to a pumping substation (PSS) and a dry cooler to evacuate heat outside the DC.
  
• Immersion Cooling (IC): the fluid is contained in a tank and cools all IT equipment in the server rather than just CPUs and GPUs; that fluid basically replaces the air circulating in OVHcloud servers thus enhancing the efficiency of any component not cooled by OVHcloud water cooling systems.

It comes with a new 3 floor server rack design, in a library format, which we can populate with up to 48 servers (1U) or 24 servers (2U), in a book format. Each server is submerged in its own tank for independent cooling, allowing large scale deployments. In addition, each server benefits from a specific monitoring system of all environmental factors ensuring safe and secure operation at the server level.

Benefits in terms of power consumption and efficiency

OVHcloud Hybrid Immersion Liquid Cooling has several advantages that makes it an efficient solution:

• The new passive rack design means there are neither pumps nor fans, resulting in zero cooling electrical consumption at the rack level
• High power racks can operate with datacentre inlet temperatures of up to 45 °C, allowing for different cooling loads at different climatic conditions
• Stay energy efficient using free cooling available on site
• The electrical consumption and CAPEX are further reduced through the elimination of the Evaporative Cooling system (no pumps) used on dry coolers for data centers located in areas where air ambient climatic temperatures are below 43 °C
• The difference between supplied and recovered temperature of DC water (DT) of 20K can be achieved through highly efficient CPU and GPU cold plates and a large thermal contact surface between the serpentine and the dielectric fluid. The patent pending serpentine design occupies a very small area within the server chassis maximizing space for components
• Global datacentre cooling infrastructure power consumption is reduced by at least 20.7% compared to the OVHcloud water cooling system
• The energy consumption of a server per year is reduced by at least 20% compared to air cooled servers and 7% versus water cooled servers
• Reduced power consumption means lower operating expenses (OPEX)

Finally, in the context of ever-increasing CPU/GPU TDP, the solution supports higher computing power density.

A new generation of datacentre

Our new Hybrid Immersion Liquid Cooling approach stands outs with several key advantages for datacentre operators:

• Actual footprint of up to 37U/m² can be guaranteed with no extra pumping and condensers systems. The design is scalable to two or three times its footprint if immersion cooling racks are installed in stacked maritime containers
• With no sealing required nor sophisticated heat exchangers and pumping circuits the capital expenditure (CAPEX) remains low
• With a datacentre outlet temperature of 65 °C, fatal heat is better captured, and affordable heat recovery systems can be envisioned
• More flexibility and freedom in design at OVHcloud, where the new proposed rack design requires access to the front side, unlike classic racks requiring access from both the front and rear sides.

Redefining datacentre indexes

With such a new game-changing technique, the usual datacentre performance indicators evolve dramatically.

Enhance PPUE^[1] to 1.004: PPUE or partial power usage effectiveness defines a certain portion of the overall PUE of a datacentre within a clearly defined boundary. Infrastructure cooling PPUE is as follows:

Reduce WUE to 0 for datacentre located in areas where ambient air climatic temperatures are below 43 °C: WUE or water usage effectiveness is a sustainability metric to measure the amount of water used by datacentres to cool IT equipment. Annual site water usage includes water used for humidification and water evaporated on-site for energy production or cooling of the datacentre and its support systems. It is defined as follow:

Environmental considerations

The Hybrid Immersion Liquid Cooling technique relies on a non-volatile dielectric hydrocarbon fluid. OVHcloud R&D tested and qualified the system with a variety of market fluids. While each fluid has different properties, we paid special attention to their compliance with the following considerations:

• Non-corrosive
• Ultra-low vaporization
• Nontoxic
• Non allergenic
• Biodegradability in 30 days
• Very high flash point
• Dielectric Strength up to 42 kV
• GWP^[2]=0
• ODP^[3]=0

Operational changes within the datacentre

The new 3 floor library format rack design is footprint optimized for a high density of servers and incidentally compute power per square meter (U/m²). It could be fitted with manual/automated guidance to extract servers, reducing the maintenance impact of one server on the overall rack.

The new Hybrid Immersion Liquid Cooling technique also improves reliability reducing failure rate by up to 60 % by eliminating risk of dust circulation. It contributes to a silent datacentre with the removal of any fan at rack and server levels. Finally, several components are no longer needed:

• No heat exchangers attached to the racks (3 heat exchangers are eliminated per rack),
• 36 fans usually installed per rack are eliminated
• All small fans inside the servers are eliminated
• Cooling modules comprised of pumps and plate heat exchangers are eliminated

What’s next?

The new OVHcloud Hybrid Immersion Liquid Cooling paves the way for unapologetically more compute power in modern datacentres. Its unique patent pending design is sustainable and preserves the opportunity to valorize fatal heat due to higher operating temperatures. Moreover, lower power and water consumption, as well as improved footprint can be achieved with the benefit of adapting to regions with harsh climatic conditions. We are incredibly excited by the very first use cases of Hybrid Immersion Liquid Cooling in the fields of banking, medical and scientific research, not forgetting gaming workloads. We can’t wait to share more with you in the future.

[1]: Partial PUE is used instead of PUE. It is expected that the global PUE will improve in a similar approach which includes the power lost in the energy distribution system [More details about the PPUE and PUE here]. In line with its transparency commitments, OVHcloud will communicate PUE values upon gathering significant data on the long run on a specific location basis.
[2]: GWP stands for Global Warming Potential. The fluid does not contribute to global warming in opposition to most of gas/liquid fluids used in cooling technologies.
[3]: ODP stands for Ozone Depletion Potential. The fluid does not contribute to ozone depletion in opposition to most of gas/liquid fluids used in cooling technologies.