In our latest installment we introduced the merits of our Hybrid Immersion Liquid Cooling Technology. This new approach to cooling essentially enables new scenarios within the data centre all the while significantly reducing power consumption by up to 20% for cooling systems dedicated to servers. Amongst other benefits, it helps for operation in harsh climatic zones. While OVHcloud’s R&D is hard at work on the next generation of cooling, it also keeps on improving the basic building blocks of cooling in the data centre. Today we’ll discuss in more detail one of the singular aspects of OVHcloud’s designed data centres that is crucial to reducing water consumption and overall cooling costs in comparison to mechanical cooling system used within data centres: Delta T.
Ever since 2003, OVHcloud has developed watercooling at scale for its servers in a free cooling approach that prohibits the use of power-hungry air conditioning thus reducing power costs and water usage. Over the years, our technology has been refined from generation to generation. All these years of work allowed us to see that one crucial but often overlooked parameter is the Delta T, a parameter that measures the difference in temperature for the water coming in and out of the data centre. From this single parameter comes greater efficiency particularly when it comes to Water Usage Effectiveness (WUE). A parameter that we continue refining with the most recent milestone being the Delta T=20K. Ready for the deep dive?
Overview of OVHcloud cooling DC principles
A typical OVHcloud data centre benefits from an indirect free cooling technique, also known as IFC, with the deployment at scale of a liquid cooling system. It helps in improving global data centre power efficiency with up to 50% energy savings compared to conventional mechanical cooling. With such a cooling strategy that leverages liquid cooling and air cooling, OVHcloud’s data centres aim at being highly efficient.
Our proprietary watercooling system recovers fatal heat generated by our servers through carefully crafted waterblocks offering direct contact with the chips. If we break down the system, there are four cooling bricks in the data centre:
- Dry Cooler (heat rejection system),
- Pumping Substation (PSS),
- Rack Rear Door Heat Exchanger (RDHX) and Cooling Distribution Unit (CDU),
- Server waterblock,
A typical water circuit in an OVHcloud data center goes like this: cold water is pumped to the rack from a pumping substation (PSS) before reaching the Rear Door Heat Exchanger (RDHX) installed at the rack level. Each rack comes with 3 RDHXs, each door having its own set of fans supporting air cooling capacity. Water then flows through the CDU or Cooling Distribution Units: placed inside the rack, CDU are comprised of pumps and plate heat exchangers (PHEXs). They distribute liquid to the waterblocks, effectively acting as a watercooling system, where the IT equipment heat will be transported back to the dry coolers located outside of the data centre for heat rejection.
Playing an important role in the cooling infrastructure, the dry coolers use external air to cool down the circulating liquid pumped from the data centre without any refrigeration. Depending on the climatic conditions, the most efficient evaporative cooling (EC) system to date is used to cool the outside air before it reaches the fins of the dry cooler.
Kelvin, how delta T is measured and why it’s a key parameter
In the research field, the usual norm is to express temperature difference in Kelvin while all temperature measurements are in °C, F or K. In this document, temperature is express in °C, while temperature difference is expressed in K. When it comes to our data centres, Delta T is the difference of water temperature between the inlet and outlet of a rack (in other words the in/out water temperature of the data center): the greater this difference is, the more efficient the data centre will be. Why? An increase in the temperature difference in the racks (dT) causes a decrease in the flow rate of water supplied per rack allowing the use of lower pumps capacity. Thus, an increase in the temperature difference permits a higher heat load per pumping unit and dry cooler for an equivalent PSS hydraulic capacity and heat rejection system.
From Delta T=5K to Delta T=20K
As we have seen, from a greater temperature difference comes more efficiency. It also means that the performance of the dry cooler is significantly improved. Indeed, water enters at higher temperature levels, while fan and evaporative cooling water consumption per kWh IT is reduced because a lower air flow rate is required. That later consequence allows the optimization of the ventilation capacity.
Since 2019, we upgraded to a new generation of cooling infrastructure for our most recent data centres capable of operating at a Delta T=20K instead of Delta T=5K. We rely on a lower number of dry coolers per data centre, yet more efficient ones, with smaller pumps and reduced piping. The result is a potential higher footprint (kW/m2) for the data centre, lower overall carbon footprint, better PPUE and an improved WUE, as well as reduced CAPEX and OPEX.
- Delta T=5K explained
In a traditional OVHcloud data centre with a Delta T=5K setting, water temperature is to be maintained at 25°C providing inlet air temperature at 30°C and inlet water temperature of the waterblock at 35°C. As shown in the figure below, two circuits in parallel are connected to the rack.
Through constant development, we improved the cooling design of our rear door heat exchanger (RDHX) to increase the inlet water temperature and reduce pinch between cold air and cold water: this way, the data center inlet water temperature is now set at 27°C against 25°C previously.
- Delta T=20K explained
Building up on the notable improvements we discussed, we were also able to setup a serial circuitry with PHEX and RDHX on the rack level in order to achieve a complete Delta T=20K environment as shown in the figure.
With water temperature maintained at 27°C, the inlet air temperature of the data centre remains at 30°C and inlet water temperature at 45°C for the waterblock. The data centre water outlet temperature is set at 47°C. On a sidenote, our robust design gives us confidence in the ability to operate with a higher temperature profile, still in a Delta T=20K setting, like 32-52°C.
What are the benefits of Delta T=20K VS Delta T=5K? And what are the advantages when applied to different data centre locations?
To measure the gains of OVHcloud’s IFC system operating with a Delta T=20K, we conducted an extensive series of benchmarks in various locations with the same Cooling Module configured first as Delta T=5K then as Delta T=20K. We should be able to highlight significant gains in terms of water consumption.
We elected for the following methodology. We collected and analyzed the weather data for the past 40 years, including temperature and relative humidity with a record every 30 minutes. For the Roubaix site for example, the warmest year happened to be 2019. Figure below shows the temperature frequency in hours extracted from climatic data recorded at Roubaix in 2019. The key hours are between 5 and 20 °C.
Based on our working principle, we can deduct the WUE in every country where we have a data centre. For your reference, in this scientific paper we estimated the theoretical WUE for some of our locations that offer the best possible representation of various climatic conditions: Roubaix (France), Mumbai (India), Singapore, VintHill (USA). For accuracy purposes we’re sharing below the estimated WUE for Roubaix:
This theoretical result is impressive. Yet, it’s important to have a better sense of what WUE can represent: we estimate that in an IFC with a Delta T=20K setting, 381 m3 of water are enough to cool a 600 kW DC versus 964 m3 in an IFC with Delta T=5K setting.
The table below is a summary of the most glaring changes between Delta T=5K and Delta T=20K:
| dT= 5K | dT= 20K | |
|---|---|---|
| Temperature profile | 25°C – 30°C @ 5K | 27°C – 47°C @ 20K |
| Water flow rate per server | Up to 0.25 l/min | Up to 0.5 l/min |
| Dry coolers consumption for 600kW of IT workload | Up to 48kW | Up to 24kW |
| Pumping Substation(s) (PSS) for 600 kW of IT workload (pump consumption) | Up to 17kW | Up to 3kW |
| Rack internal pumping (CDU) | 0 | Up to 7kW for 600kW of IT workload |
| PSS footprint | 125 kg/pump | 50 kg /pump |
| Average annual WUE (l/kWh) | 0.16 | 0.09 |
Versus the competition?
To date, there are two different families of water liquid cooling techniques employed within data centres: close-coupled cooling and direct liquid cooling (see table). Watercooling solutions take advantage of the heat-carrying capacity of the water that is 3,3 times higher than that of air for a dramatic reduction in thermal resistance. With CPU and GPU being the most power-hungry parts in a server, liquid cooling via direct-to-chip contact contributes in returning warmer water from IT equipment than that of closely coupled cooling systems. Logically, the coolant temperature directly impacts energy savings and cooling efficiency participating in a significant decrease in the total DC site energy consumption by 14 to 20%1. Please note, more information on the above can be obtained here.
| Liquid cooling solutions | Description | |
|---|---|---|
| Close coupled cooling with a typical inlet water temperature = 5 to 23 °C | In-row Cooling | Row-based air conditioning units are installed inside the rack rows |
| In-rack Cooling | The cooling system is combined with the server rack, and both are sealed completely. | |
| Direct-to-chip cooling with a typical inlet water temperature 27–45 °C | Rear Door Heat Exchanger | Based on a combination of fans and a finned heat exchanger installed on the rear side of racks. |
| Cold plates (waterblocks) | A cold plate with micro-channels in direct contact with components like CPUs and GPUs. Here, the water flowing across absorbs chip energy. | |
What’s next?
As we celebrate the 20th anniversary of our watercooling technology this year, we continue to innovate. We have been working on designing a new breed of data centre employing climate friendly technologies when it comes to cooling. In a global context of surging energy prices and several water shortages in the 2022 summer, it becomes quite obvious as to why our previous technological bets pay off. With best-in-class PUE and WUE for our data centres, we continue to work tirelessly on improving our footprint within the ecosystem illustrating our deep commitment towards sustainability. Hopefully by now you have come to realize how important to the data centre is the Delta T parameter. From the infrastructure point of view to the tangible water efficiency, Delta T=20K contributes in adapting to various geographies and ultimately water conservation.
The water consumption of data centres has indeed a tremendous impact locally going that far as to affect river flow, groundwater levels and influence overall water resources for a whole territory. Thus, reducing WUE proves critical for any modern data centres to lower the overall impact on local communities.
The impressive saving of water illustrated by our WUE also contributes to a streamlined PUE for compliance with ISO/IEC 30134 standard. Last, but not least, the demonstrated advantages of Delta T=20K will prove to be a solid basis for the next generation of improvements our teams develop.
1: Performance analysis of new liquid cooling topology and its impact on data centres – https://www.sciencedirect.com/science/article/abs/pii/S1359431122006779?via%3Dihub
Ali Chehade is a Doctor of Engineering in Fluid Mechanics and Thermal Energy. With 12 years of experience in R&D/Engineering, he has worked in the innovation of DC cooling systems and industrial facilities. He has led projects dealing with their thermal performance and energy saving and has implemented techniques/strategies to make them environmentally friendly. Within OVHcloud Ali, as head of R&D-Cooling for 7 years and surrounded by a dozen of engineers, promotes and coordinates the development of solutions to improve the construction and energy performance of DCs.
https://www.linkedin.com/in/ali-chehade
Mohamad is a Doctor of Engineering specializing in Thermal and Energy. Mohamad has an R&D and Engineering Background, gave him the opportunity to execute vital projects in several technologies during solid years of practice experience.
His areas of expertise includes heat transfer process, hardware components and software, industrial products, electronics cooling systems, data centers thermal performance and energy saving analysis.
Julien is a Communications and Public Relations Manager at OVHcloud. With a number of years of experience in PR, Julien is a long-time technology enthusiast and former professional tech journalist. With more than 20 years of experience in the IT industry, he witnessed his share of IT cycles, led various teams and wrote quite a number of papers. His areas of expertise include hardware components and software.