The building infrastructure of a data center is the physical and structural part of the data center that houses the computing infrastructure. It includes the building, the power supply, the cooling system, and the failure recovery system.
Power systems
In order to protect the data center against power failure, the housing building must be equipped with many different types of power systems (for redundancy). The most common power systems are:
- Outside power supply: the data center is connected to the electrical national system through a high-voltage line, and distributed via power distribution devices placed outside the building. That electrical power is used to power up the whole data center and to charge the backup batteries
- Diesel generators: in case of power failure, the data center could rely on diesel generators to keep the data center running. The diesel generators are usually placed outside the building (due to their noise, pollution and safety reasons) and are connected to the power distribution devices. The diesel generators are usually used to power up the data center until the power supply is restored.
- Batteries: the data center is equipped with batteries to provide power in case of failure that lasts for a short period of time (e.g., a few minutes). The batteries are usually used to power up the data center until the diesel generators are started and connected to the power distribution devices. They are also used to start the diesel generators in case of power failure.
- Rotary UPS Systems: they store electrical energy in the form of kinetic energy. The rotary UPS systems are composed of a motor/generator that is connected to a flywheel. When the power supply is available, the motor/generator is used to spin the flywheel and store energy in the form of kinetic energy and the flywheel is used to generate electrical energy.
Cooling systems
The cooling system of a data center is usually a very expensive component of the data center, and it is composed of coolers, heat-exchangers, and cold water tanks. The IT equipment generates a lot of heat, and the cooling system is used to remove that heat from the data center. The cooling system is usually composed of two main components:
- Heat exchangers: they are used to remove the heat generated by the IT equipment. Usually placed outside the building and close to the IT equipment, are used to cool down the hot air generated by the IT equipment.
- Cold water tanks: they are used to store cold water that is used to cool down the hot air generated by the IT equipment.
- Turbo coolers: they are used to cool down the hot air generated by the IT equipment.
Loops
The simplest way to cool down the data center is to open the windows and let the fresh air cool down the data center. This is an open-loop system. The open-loop system is the simplest topology, but it is not the most efficient. In this case, the fresh air, coming from the outside and filtered to remove dust and other particles, is used to cool down hot air discharged by the IT equipment. This is a free cooling system, because it the cold outside air to either help the production of chilled water or directly cool servers. It is not completely free in the sense of zero cost, but it involves very low-energy costs compared to chillers.
The closed-loop system is more complex and more efficient than the open-loop system. The goal is to isolate and remove heat from the servers and transport it to a heat exchanger. Cold air flows to the servers, heats up, and eventually reaches a heat exchanger to cool it down again for the next cycle through the servers. The closed-loop system is composed of two loops: the first loop is the airflow through the underfloor plenum, the racks, and back to the CRAC (a 1960s term for computer room air conditioning); the second loop leads directly from the CRAC to external heat exchangers (typically placed on the building roof) that discharge the heat to the environment. Inside the IT room, the servers are placed back-to-back in rows: this creates hot corridors (that need to be cooled down) and cold corridors (that need to be kept cold).
In huge datacenters, a three-loop system is used: in the first loop (inside the data center) the hot air coming from the IT equipment is cooled down by fans and heat exchangers; in the second loop, the hot air coming from the first loop is cooled down by the cooling plant using cold water; in the third loop, the hot water coming from the second loop is cooled down using chiller condensers and evaporative cooling towers. The cooling towers cool a water stream by evaporating a portion of it into the atmosphere. They do not work as well in very cold climates, because they need additional mechanisms to prevent ice formation.
Each topology presents tradeoffs in complexity, efficiency, and cost:
- Fresh air cooling can be very efficient but does not work in all climates, requires filtering of airborne particulates, and can introduce complex control problems.
- Two-loop systems are easy to implement, relatively inexpensive to construct, and offer isolation from external contamination, but typically have lower operational efficiency.
- A three-loop system is the most expensive to construct and has moderately complex controls, but offers contaminant protection and good efficiency.
Server cooling
Server are placed in racks, where in-rack cooler can be added to cool down the hot air generated by the servers. The in-rack cooler adds an air-to-water heat exchanger at the back of a rack so the hot air exiting the servers immediately flows over coils cooled by water, essentially reducing the path between server exhaust and CRAC input. In-row cooling works like in-rack cooling except the cooling coils aren’t in the rack, but adjacent to the rack.
Another way to cool down the servers is to use liquid cooling. We can directly cool server components using cold plates, using local liquid-cooled heat sinks. It is impractical to cool all compute components with cold plates, so only the components with the highest power dissipation are targeted for this type of cooling, while other components are chilled only using air-cooling. The liquid circulating through the heat sinks transports the heat to a liquid-to-air or liquid-to-liquid heat exchanger that can be placed close to the tray or rack, or be part of the data center building (such as a cooling tower). Liquid cooling is used in some specific cases, like the TPUs or the GPUs used in some data centers.
Building infrastructure
The building infrastructure of a data center is the physical and structural part of the data center that houses the computing infrastructure. It includes the building, the power supply, the cooling system, and the failure recovery system.
The classic data center building is huge and is composed of many different rooms, each one dedicated to a specific task. The building is usually composed of a main room where the servers are placed, a room where the power supply is placed, a room where the cooling system is placed, and a room where the failure recovery system is placed.
A more recent approach to data center building is the container-based data center. In this case, the servers are placed inside a container (typically 6 to 12 meters long) and integrated with the cooling system and the power distribution system. In this way, the datacenter is standardized and more efficient than the classic data center, because it is easier to cool down and to power up. It’s also easier to scale up and down the data center, because it is easier to add or remove containers and this modular approach is used even in standard data centers to add more computing power, while keeping the cooling and power supply systems unchanged. It’s also used when more computing power is needed in a short time.
Power consumption
Data center power consumption is an issue, since it can easily reach several MWs. Each component in the building infrastructure consumes power. The cooling system usually requires about half the energy required by the IT equipment. This is due to the high level wasted energy in the energy transformation process.
Definition
The energy transformation process is the process that transforms the electrical energy coming from the power supply into the energy used by the IT equipment.
This process is not 100% efficient, and a lot of energy is wasted in the form of heat, that must be removed from the data center using the cooling system.
| Infrastructure | Component | Power consumption |
|---|---|---|
| Servers | CPU, memory, disks | 45% |
| Infrastructure | UPS, cooling, power distribution | 25% |
| Power draw | Electrical utility costs | 15% |
| Network | Switches, links, transits | 15% |
Definition
The Power usage effectiveness (PUE) is a metric that can be used to measure the efficiency of a data center. It is the ratio of the total amount of energy used by a DC facility to the energy delivered to the computing equipment.
where the
covers IT systems (servers, network, storage) and other equipment (cooling, UPS, switch gear, generators, lights, fans, etc.).
The PUE is not a one-fits-all metric for energy efficiency for servers, networking and storage in the data center. It’s not even a constant value, because it changes over time, depending on the energy consumption of the data center and the workload of the IT equipment (during the day, the energy consumption of the data center is higher than during the night, during the cooling season the energy consumption of the data center is less than during the heating season, etc.)
Nowadays, data centers are built to be more energy efficient, using renewable energy sources, like solar panels or wind turbines, and using more efficient cooling systems, like the free cooling system. In most of the modern data centers, the PUE is usually around 1.8, but some data centers have a PUE of 1.2 or even lower (meaning that the data center is very energy efficient).
| PUE | DCiE | Level of efficiency |
|---|---|---|
| 1.2 | 0.83 | Very efficient |
| 1.5 | 0.67 | Efficient |
| 2.0 | 0.50 | Average |
| 2.5 | 0.40 | Inefficient |
| 3.0 | 0.33 | Very inefficient |
A data center can be classified in four different tier levels, based on the availability of the data center. The tier level is a measure of the availability of the data center, and it is based on the number of hours per year that the data center is available. The tier levels are:
| Tier Level | Availability | Downtime/Year | Requirements |
|---|---|---|---|
| Tier 1 | 99.671% | 28.8 hours | Single non-redundant distribution path serving the IT equipment, non-redundant capacity components, basic site infrastructure |
| Tier 2 | 99.741% | 22.5 hours | Meets or exceeds all Tier 1 requirements, redundant site infrastructure capacity components |
| Tier 3 | 99.982% | 1.6 hours | Meets or exceeds all Tier 1 and Tier 2 requirements, multiple independent distribution paths serving the IT equipment, all IT equipment must be dual-powered and fully compatible with the topology of the site’s architecture, concurrently maintainable site infrastructure |
| Tier 4 | 99.995% | 0.8 hours | Meets or exceeds all Tier 1, Tier 2, and Tier 3 requirements, all cooling equipment is independently dual-powered, including chillers and heating, ventilating, and air-conditioning (HVAC) systems, fault-tolerant site infrastructure with electrical power storage and distribution facilities |