Server immersion cooling


Immersion cooling is an IT cooling practice by which IT components and other electronics, including complete servers, are submerged in a thermally conductive dielectric liquid or coolant. Heat is removed from the system by circulating liquid into direct contact with hot components, then through cool heat exchangers. Fluids suitable for immersion cooling have very good insulating properties to ensure that they can safely come into contact with energized electronic components.

Dielectric liquids

In general, there are two main liquid categories, Hydrocarbons and fluorocarbons. Dielectric liquids are divided into single- and two-phase applications.
"Enclosed chassis" require dripless connectors to interface to the individual chassis. These chassis are usually based on traditional rack style implementations. The dripless connectors usually require a small closed circuit cooling loop with a coolant to protect the flow integrity through relatively small pipes and connectors. The closed circuit is facilitated by a CDU or Coolant Distribution Unit, which usually can facilitate multiple racks at once.
“Open bath” refers to the “open” liquid-air interface and thus surface tension between the liquid and the air being a distinctive element. Open bath systems are usually tanks which contain a larger body of dielectric liquid where electronics are immersed into the bath. Multiple electronic assemblies are sharing the same liquid. This liquid can be based on single- or two-phase technology. Regardless of the term, open bath systems can be fully sealed, but are always opened from the top to service IT equipment. The water circuit for open bath immersion systems is either connected to a CDU which circulates the dielectric liquid, or to an integrated heat exchanging device which is part of the tank. For a facility interface, CDU's are usually designed for 100 kW or more, whereas an integrated heat exchanging device is usually designed for 10-100 kW cooling capacity.
"Hybrid" refers to combinations of the forementioned shapes and forms. There are numerous solutions in the industry already with unique and diverse approaches. These may not always fit in any designated definition. Depending on which aspect is relevant, these strategies may be positioned as any overlapping definition.

Evolution

Immersion cooling has the potential of becoming a popular server cooling solution, in particular for green data centers, as it allows operators to drastically reduce their energy usage through the elimination of the expensive air cooling infrastructure including on-board server fans, CRACs, A/C compressors, air-circulation fans, necessary duct work, air handlers, and other active ancillary systems such as dehumidifiers. These systems are replaced with efficient low speed liquid circulation pumps and heat exchanger and/or dry cooler systems. The power reduction that can be achieved through the use of immersion cooling is often measured in terms of Power Usage Effectiveness. However, It is important to note that the definitions of PUE for air-cooled devices and liquid immersion cooled devices differs in one important aspect, which makes such direct comparisons misleading in favor of air cooling.
This discrepancy in the definition of PUE for the different cooling methods results in the PUE of air-cooled data centers generally being overstated by anywhere from 10-20% when compared against the PUE of a liquid immersion cooled facility of the same power usage.
Servers and other IT hardware cooled by immersion cooling do not require fans to circulate the dielectric liquid, thus they are removed from the system prior to immersion. Thermal pastes which are typically used on heat spreaders for CPUs and other chips may require replacement with a different compound in order to avoid the thermal degradation within the dielectric liquid. Depending on the type of application, solder, Indium foil and thermally conductive epoxies may be used as a replacement materials.
The temperatures used in immersion cooling are determined by the highest temperature at which the devices being immersed can reliably operate. For servers this temperature range is typically between, however in ASIC based crypto mining devices this range is often extended up to 75C. This increase in the high end of the temperature range allows data centers operators to use entirely passive dry coolers, or much more efficient evaporative or adiabatic cooling towers instead of chiller-based air cooling or water chillers. This increase in the temperature range also allows operators using Single-phase Immersion Coolants to more effectively use the change in outdoor temperatures to get more efficient cooling from their systems because the single-phase systems are not limited in their effectiveness by the boiling point of the coolant as is the case with two-phase coolants.
Multiple relevant brands like Intel and Facebook have already validated the advantages of submerging servers.
Current commercial applications for immersion cooling range from datacenter oriented solutions for commodity server cooling, server clusters, HPCC applications and Cryptocurrency mining. and mainstream cloud-based and web hosting architectures. Electric vehicle and battery manufacturers also employ liquid immersion cooling in batteries, drive-train, Kinetic energy recovery systems, electric motors, electric motor controllers, and other on-board electronic sub systems. Liquid immersion cooling is also used in the thermal management of LEDs, Lasers, X-Ray machines, and Magnetic Resonance Imaging devices.
A different industry where immersion cooling is applied to electronic components is deep sea research where Remotely Operated Underwater Vehicles with electronic equipment are filled with Single-phase Liquid Dielectrics to both protect them from corrosion in seawater and as a pressure compensating fluid to avoid being crushed by the extreme pressure exerted on the ROV while working in the deep sea. This application also includes the cooling of the electric motors used for under sea propulsion.
Until about 2014, the technology was typically only utilized in special very intensive supercomputing projects, similar to the Cray Computer Applications. However, since the dramatic increase in global energy consumption by data centers, there is now a focus on energy efficiency which has driven the utilizing of liquid immersion cooling in both data centers and crypto mining operations to reevaluate its application. In addition the advent of new very high density CPUs and GPUs for use in real-time processing, Artificial intelligence, machine learning, and data mining operations is pressuring users and data center operators to evaluate liquid immersion cooling for ability to cool high density racks as well as reduce the total mechanical footprint of data centers.
Since 2005, several companies tried to claim patents based on the open bath approach with external pump installations. Most, if not all of these claims have been denied on the basis that IBM had already patented the technology.

History

19th and 20th century immersion milestones:
21st century immersion milestones:

Open bath immersion cooling

Open bath immersion cooling is a data center cooling technique that implies fully submerging IT equipment in dielectric liquid. The “open” aspect does not refer to an open or sealed system, but refers to the “open” liquid-air interface and thus surface tension between the liquid and the air being a distinctive element.
These baths allow the coolant fluid to be moved through the hardware components or servers submerged in it.
Single phase immersion requires circulation of the dielectric liquids by pumps or by natural convection flow. These liquids always remain in liquid state while operating. They never boil or freeze. The dielectric coolant is either pumped through an external heat exchanger where it is cooled with any facility coolant, or the facility coolant is pumped through an immersed heat exchanger, which facilitates heat transfer within the dielectric liquid.
In two-phase systems, fluorocarbons are used as heat transfer fluids. Heat is removed in a two-phase system, where the liquid literally boils when it comes in contact with hot components, due to its low boiling point. The system takes advantage of a concept known as “latent heat” which is the heat required to change the phase of a fluid, this occurs when the two-phase coolant comes in contact with the heated electronics in the bath that are above the coolants boiling point. Once the two-phase coolant enters its gas phase it must be cooled or condensed, typically through the use of water cooled coils placed in the top of the tank. Once condensed the two-phase coolant drips back into the primary cooling tank. The two-phase coolant in the tank generally remains at it “saturation temperature”. Energy transferred from the servers into the two-phase coolant will cause a portion of it to boil off into a gas. The gas rises above the liquid level where it contacts a condenser which is cooler than the saturation temperature. This causes the gaseous state coolant to condense back into a liquid form and fall back into the bath.

Enclosed chassis immersion cooling

A different approach to open bath immersion cooling are sealed server immersion cooling solutions, where servers are specially built in a liquid-tight casing. The dielectric coolant is circulated inside or pumped through each server to collect heat from the components. The heat is then taken to a manifold or backplane in the rack where it is either circulated directly outside the building to a cooling tower or heat exchanger or cooled directly at the rack with a facility coolant infrastructure.
The main advantage of this approach is that servers are mounted in self contained vessels that can be easily replaced in the rack and are not required to be placed in horizontal baths. The most common disadvantage is that not all hardware can be used as the vendor defines the hardware specs of the sealed servers.