2008 IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing

A measurement-based method for improving data center energy efficiency

Hendrik F. Hamann
IBM T.J Watson Research Center
hendrikh@us.ibm.com

Abstract identify current problems and help to mitigate the

In this paper we discuss a novel sensing and
measurement technology which allows visualizing the
3D temperature and heat distributions in a data center
or other complex thermal system. Specifically, we
leverage the high spatial resolution of IBM's Mobile
Measurement Technology (MMT). We show that
substantially improvements in energy efficiency can be
obtained by the MMT method.
1. Introduction
The increase of power consumption of IT
equipment is imposing significant strains on data
centers (DC) the supporting infrastructure [1]. Today,
DC managers are struggling with to balance
maintaining the inlet temperatures for each equipment
(within the required specifications) and saving energy
as well as capital cost associated with cooling.
Consequently, a much better understanding and
management of the thermal and energy distribution
within the DC is essential, in particular considering
that up to 50 % of the total DC energy consumption
can be governed by the cooling.
Today, extensive finite element models are
deployed to understand hotspots and cooling solutions
in data centers. These models solve a complicated set
of coupled partial differential equations (i.e., Navier
Stokes and heat conduction/convection equation) using
computational fluids dynamics (CFD) techniques.
While it is very difficult to accurate represent the
physical dimensions and conditions of the DC, such
computations are often difficult and lengthy due to the
multi-dimensionality of the modeling problem with its
various lengths and time scales. Furthermore, current
model predictions have not sufficiently been validated
[x]. Ironically and despite the importance of DC
thermal management there has been a complete lack of
detailed, experimental data, which would readily
cooling challenges of today's DCs
2. Mobile Measurement Technology
In order to address these shortcomings and the
paucity of detailed temperature data in DCs, a simple
experimental technique was recently introduced, which
allows for fast 3D mapping throughout a large scale
computing system [2]. As illustrated in Fig.1 a sensor
network is mounted at a regular distance on the mobile
measurement cart, which is readily moved through an
actual data center while temperature data is being
logged from all the sensors as a function of X, Y and Z
coordinates. The sensor distance is 8'' in the lateral
direction and 12'' in the z-direction. We show here a
“T”-shaped cart which allows for thermal
measurements above the racks. The cart design is
modular can be adjusted to different heights and
shapes. By repeating the measurements throughout the
DC and in combination with the cart’s position and
orientation we can digitize the three dimensional DC
space.
Fig.1: Illustration of the MMT in a raised floor data
center.
For example we can construct a 3-D temperature or
heat map of an actual data center. Fig.2 shows

978-0-7695-3158-8/08 $25.00 © 2008 IEEE 312

DOI 10.1109/SUTC.2008.65
exemplary data from a raised-floor DC, which was
scanned using the Mobile Measurement.
Fig.2: 3D thermal distribution of a cold aisle, which
was scanned using the MMT.
Specifically, in Fig.2 we show results from a "cold
aisle" with the server inlets being fed by the cold air
from the perforated tiles. The backside of the server is
referred to a "hot aisle" where the heated air from the
servers is exhausted. As it is evident from Fig.2 not all
servers within the racks are supplied with the same
inlet temperature. In fact, a closer look reveals that a
few servers suffer from so called hotspots. These
hotspots are often responsible for major energy
inefficiencies, because an (energy-costly) solution to
such hotspots involves compensating for these hotspots
by chosen a lower chiller set point, which drives
disproportionately up the energy costs for the DC. It
has been shown that is useful to distinguish between
temperature variations between racks (inter-rack or
horizontal hotspots) and across the same rack (intra-
rack or vertical hotspots) [2].
Fig.3: 2D MMT measured thermal distribution of a
DC at 5.5 feet height.
In the following these hotspot effects are quantified.
In Fig.3 we show an MMT measured 2D temperature
distribution at a height of5.5 feet for twelve server
racks (grey boxes). Each rack contains 12 servers (total
144 servers). In Fig.4 we have plotted the inlet
temperatures of these 12 racks from Fig.3 as a function
of height. For reference we show the server rack next
313
to the plot (shown in scale). It is evident from Fig.4
that we have enormous temperature variations across
the rack starting at 15oC for servers at the bottom of
the rack up to 35oC for servers at 5.5 feet height.
Clearly, a variable frequency fan within the server rack
or better air flow provisioning could readily prevent
these hotspots improving the energy efficiency of the
DC. Fig.4 also shows large inter-rack hotspots. For
example, rack #1,6,7,12 show much larger temperature
variations than some of the other server racks.
Evidently, these racks are located at the end of the aisle
and thus they are more prone for recirculation effects.
Fig.4: MMT measured inlet temperatures as a function
of height for 12 server racks (see Fig.3).
3. Conclusion
In summary, we have presented a simple
experimental methodology for rapid 3D thermal
mapping of large data centers. This new technique not
only allows for benchmarking current data center
models but it also enables the rapid diagnosis of
existing cooling problems within the data center. In
this case study we showed hotspots due to intermixing
between cold and hot air which suggest that the current
cooling scheme can be significantly improved.
Acknowledgement: HFH thanks the whole IBM
MMT team for contributions.
4. References
[1] J.G. Koomey, "Estimating Total Power
Consumption By Servers In The US and The World",
A report by the Lawrence Berkeley National
Laboratory, February, 2007.
[2] H. F. Hamann ,J. Lacey, M. O’Boyle, R. Schmidt,
and M. Iyengar , Rapid 3D thermal imaging of large-
scale computing facilities, IEEE CPMT, vol.xx, no. x.,
March 2008 (accepted) - see also US patent application
US20070032979A1