If you think about it, datacentre cooling infrastructure hasn’t changed much over the past few decades. Whilst compute power and efficiency have remarkably improved over the years, many datacentres still rely on racks whose cooling depends on a variety of systems comprised of fans. Heat, especially fatal heat, remains a top concern to keep servers up-and-running. Whilst some datacentres have opted for power hungry solutions such as air conditioning (direct expansion), others have a free cooling approach.
Here at OVHcloud, we pioneered free cooling, and have used water cooling at scale to cool down our servers since 2003. This unique approach helped us to set high standards in PUE/WUE indexes for our global datacentres, all the while lowering our overall carbon impact.
With the evolution of IT equipment power, OVHcloud R&D worked on a new generation of datacentre cooling solutions. One of the most promising technologies today is immersion cooling, thanks to direct contact of the fluid with hardware components. It allows us to build datacentres all over the world maintaining low PUE/WUE indexes even in harsh climatic zones.
In this blog post, we will explain how we developed our own breed of immersion cooling to achieve several goals, from improving datacentre footprint and reducing power consumption, to adapting to new operating conditions and improved overall reliability.
OVHcloud Hybrid Immersion Liquid Cooling
Immersion Cooling is the practice of submerging electronic devices in a thermally and not electrically conductive liquid.
The typical immersion cooling solution usually employs pumps, heat sink structures, heat exchangers, condensers, sealed evaporative equipment etc. that either consume large amounts of energy to operate, require sealed casings, or occupy relatively large surface areas that limit the number of servers that can be implemented.
OVHcloud developed its own Hybrid Immersion Liquid Cooling technique. It is comprised of a direct to chip water cooling system and a passive natural single phase immersion cooling system containing two fluids:
- Water: cooling a heat sink through water blocks disposed on CPUs and GPUs with the same solution used in all OVHcloud servers and a proprietary serpentine convection coil connected to a pumping substation (PSS) and a dry cooler to evacuate heat outside the DC.
- Immersion Cooling (IC): the fluid is contained in a tank and cools all IT equipment in the server rather than just CPUs and GPUs; that fluid basically replaces the air circulating in OVHcloud servers thus enhancing the efficiency of any component not cooled by OVHcloud water cooling systems.
It comes with a new 3 floor server rack design, in a library format, which we can populate with up to 48 servers (1U) or 24 servers (2U), in a book format. Each server is submerged in its own tank for independent cooling, allowing large scale deployments. In addition, each server benefits from a specific monitoring system of all environmental factors ensuring safe and secure operation at the server level.
Benefits in terms of power consumption and efficiency
OVHcloud Hybrid Immersion Liquid Cooling has several advantages that makes it an efficient solution:
- The new passive rack design means there are neither pumps nor fans, resulting in zero cooling electrical consumption at the rack level
- High power racks can operate with datacentre inlet temperatures of up to 45 °C, allowing for different cooling loads at different climatic conditions
- Stay energy efficient using free cooling available on site
- The electrical consumption and CAPEX are further reduced through the elimination of the Evaporative Cooling system (no pumps) used on dry coolers for data centers located in areas where air ambient climatic temperatures are below 43 °C
- The difference between supplied and recovered temperature of DC water (DT) of 20K can be achieved through highly efficient CPU and GPU cold plates and a large thermal contact surface between the serpentine and the dielectric fluid. The patent pending serpentine design occupies a very small area within the server chassis maximizing space for components
- Global datacentre cooling infrastructure power consumption is reduced by at least 20.7% compared to the OVHcloud water cooling system
- The energy consumption of a server per year is reduced by at least 20% compared to air cooled servers and 7% versus water cooled servers
- Reduced power consumption means lower operating expenses (OPEX)
Finally, in the context of ever-increasing CPU/GPU TDP, the solution supports higher computing power density.
A new generation of datacentre
Our new Hybrid Immersion Liquid Cooling approach stands outs with several key advantages for datacentre operators:
- Actual footprint of up to 37U/m2 can be guaranteed with no extra pumping and condensers systems. The design is scalable to two or three times its footprint if immersion cooling racks are installed in stacked maritime containers
- With no sealing required nor sophisticated heat exchangers and pumping circuits the capital expenditure (CAPEX) remains low
- With a datacentre outlet temperature of 65 °C, fatal heat is better captured, and affordable heat recovery systems can be envisioned
- More flexibility and freedom in design at OVHcloud, where the new proposed rack design requires access to the front side, unlike classic racks requiring access from both the front and rear sides.
Redefining datacentre indexes
With such a new game-changing technique, the usual datacentre performance indicators evolve dramatically.
Enhance PPUE[1] to 1.004: PPUE or partial power usage effectiveness defines a certain portion of the overall PUE of a datacentre within a clearly defined boundary. Infrastructure cooling PPUE is as follows:
Reduce WUE to 0 for datacentre located in areas where ambient air climatic temperatures are below 43 °C: WUE or water usage effectiveness is a sustainability metric to measure the amount of water used by datacentres to cool IT equipment. Annual site water usage includes water used for humidification and water evaporated on-site for energy production or cooling of the datacentre and its support systems. It is defined as follow:
Environmental considerations
The Hybrid Immersion Liquid Cooling technique relies on a non-volatile dielectric hydrocarbon fluid. OVHcloud R&D tested and qualified the system with a variety of market fluids. While each fluid has different properties, we paid special attention to their compliance with the following considerations:
- Non-corrosive
- Ultra-low vaporization
- Nontoxic
- Non allergenic
- Biodegradability in 30 days
- Very high flash point
- Dielectric Strength up to 42 kV
- GWP[2]=0
- ODP[3]=0
Operational changes within the datacentre
The new 3 floor library format rack design is footprint optimized for a high density of servers and incidentally compute power per square meter (U/m2). It could be fitted with manual/automated guidance to extract servers, reducing the maintenance impact of one server on the overall rack.
The new Hybrid Immersion Liquid Cooling technique also improves reliability reducing failure rate by up to 60 % by eliminating risk of dust circulation. It contributes to a silent datacentre with the removal of any fan at rack and server levels. Finally, several components are no longer needed:
- No heat exchangers attached to the racks (3 heat exchangers are eliminated per rack),
- 36 fans usually installed per rack are eliminated
- All small fans inside the servers are eliminated
- Cooling modules comprised of pumps and plate heat exchangers are eliminated
What’s next?
The new OVHcloud Hybrid Immersion Liquid Cooling paves the way for unapologetically more compute power in modern datacentres. Its unique patent pending design is sustainable and preserves the opportunity to valorize fatal heat due to higher operating temperatures. Moreover, lower power and water consumption, as well as improved footprint can be achieved with the benefit of adapting to regions with harsh climatic conditions. We are incredibly excited by the very first use cases of Hybrid Immersion Liquid Cooling in the fields of banking, medical and scientific research, not forgetting gaming workloads. We can’t wait to share more with you in the future.
[1]: Partial PUE is used instead of PUE. It is expected that the global PUE will improve in a similar approach which includes the power lost in the energy distribution system [More details about the PPUE and PUE here]. In line with its transparency commitments, OVHcloud will communicate PUE values upon gathering significant data on the long run on a specific location basis.
[2]: GWP stands for Global Warming Potential. The fluid does not contribute to global warming in opposition to most of gas/liquid fluids used in cooling technologies.
[3]: ODP stands for Ozone Depletion Potential. The fluid does not contribute to ozone depletion in opposition to most of gas/liquid fluids used in cooling technologies.
Ali Chehade is a Doctor of Engineering in Fluid Mechanics and Thermal Energy. With 12 years of experience in R&D/Engineering, he has worked in the innovation of DC cooling systems and industrial facilities. He has led projects dealing with their thermal performance and energy saving and has implemented techniques/strategies to make them environmentally friendly. Within OVHcloud Ali, as head of R&D-Cooling for 7 years and surrounded by a dozen of engineers, promotes and coordinates the development of solutions to improve the construction and energy performance of DCs.
https://www.linkedin.com/in/ali-chehade
Mohamad is a Doctor of Engineering specializing in Thermal and Energy. Mohamad has an R&D and Engineering Background, gave him the opportunity to execute vital projects in several technologies during solid years of practice experience.
His areas of expertise includes heat transfer process, hardware components and software, industrial products, electronics cooling systems, data centers thermal performance and energy saving analysis.
Julien is a Communications and Public Relations Manager at OVHcloud. With a number of years of experience in PR, Julien is a long-time technology enthusiast and former professional tech journalist. With more than 20 years of experience in the IT industry, he witnessed his share of IT cycles, led various teams and wrote quite a number of papers. His areas of expertise include hardware components and software.