Вы находитесь на странице: 1из 72

Energy-Proportional Computing:

A New Definition
David A. Wood
Professor, Computer Sciences

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

Outline
Background & Motivation
New ideals
New metrics
Management
Summary

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

Datacenter Energy Consumption


U.S. data center energy [NRDC, Anthesis]
2013: 91B KWh 34 coal-fired 500-MW power plants, 1 yr
2020: 140B KWh 50 coal-fired 500-MW power plants , 1 yr
$13B/yr electricity costs

Typical datacenter
energy consumption
breakdown
[Dimension Data]

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

PUE: Power Usage Effectiveness


Building Load
Power
Total
Facility
Energy

IT Equipment
Energy
Cooling

PDUs
Switchgears
Generators
Chillers
UPS
Pumps

CRAHs
Misc
CRACs
DX
Lighting

Total Facility Energy


PUE =
IT Equipment Energy 1
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

IT Load
Servers
Storage
Networks
Monitors
Workstations
Laptops

[The Green Grid]

Datacenter Energy Consumption


U.S. data center energy [NRDC, Anthesis]
2013: 91B KWh 34 coal-fired 500-MW power plants, 1 yr
2020: 140B KWh 50 coal-fired 500-MW power plants , 1 yr
$13B/yr electricity costs

Typical datacenter
energy consumption
breakdown
[Dimension Data]

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

PUE: Power Usage Effectiveness

Total Facility Energy


PUE =
IT Equipment Energy 1
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

[The Green Grid]

PUE: Power Usage Effectiveness


PUE > 1 energy waste due to non-IT infrastructure
Strong impact in reducing non-IT energy waste
PUE estimates for current datacenters
Allied Control: 1.02, Facebook: 1.08, Google: 1.08, avg. 1.12,
Green IT Cube: 1.07, Microsoft: 1.125, OVH: 1.09
Average: 1.7
Missing: IT equipment energy waste
This talk focuses on server energy

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

Energy Efficiency
Work Performance
=
Energy =
Power
Performance = Load served
Examples: BIPS/Watt, ssj_ops/Watt, Instructions/nJ,
max Emin

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

EP: Energy Proportionality


Work Performance
=
Energy =
Power
EP: Energy Work
[eliminates energy waste ]
equiv., Power Performance
equiv., constant
First proposed by Barroso and Hlzle, 2007
Does EP imply Emin? Converse? Identical goals?

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

Conventional Wisdom
We see that peak energy efficiency occurs at

peak utilization and drops quickly as utilization


decreases. Luiz Andr Barroso and Urs Hlzle, IEEE
Computer, 2007
Energy waste at low utilizations/loads
Servers are typically 10 50% loaded

EP as a primary design goal


Strong impact
Low idle power, wide dynamic power range
Later, Dynamic EP [Lo et al., ISCA 2014]
Accounts for unavoidable idle power
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

10

Example: SpecPower on Haswell


Java-based load-varying workload
3 calibration intervals to detect max. load (100%)
Measurement interval Loads: 100%, 90%, 80%, , 10%, Idle
Performance: transactions per second
Power: system power

Quad-core (eight-thread) Intel Haswell server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

11

EP on Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

12

EP on Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

13

EP on Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

14

EP on Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

15

Results from 2014

[Lo, et al. ISCA 2014]

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

16

EP on Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

17

EP on Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

18

Efficiency of Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

19

Efficiency of Conventional Server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

20

(more) Conventional Wisdom


In an energy-proportional system, explicit power

management is unnecessary, as power consumption


varies naturally with utilization. Meisner, Gold, Wenisch,
PowerNap: eliminating server idle power, ASPLOS 2009

Ideally systems would exhibit energy-

proportionality, wherein servers consume power in


proportion to their load. Meisner, Sadler, Barroso, Weber,
Wenisch, Power management of online data-intensive services, ISCA 2011

...; modern servers are only maximally efficient at

100%. Meisner, Wenisch, Does low-power design imply energy


efficiency for data centers?, ISLPED 2011
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

21

Current (Reconfigurable) Systems


Reconfigurable resources
DVFS
Number of cores
Threads per core
Prefetching
Cache size

Multiple configurations may serve the same load
Different power & energy consumptions

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

22

EP on Reconfigurable Systems

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

23

EP on Reconfigurable Systems

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

24

EP on Reconfigurable Systems

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

25

EP on Reconfigurable Systems

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

26

Pareto Frontier
Pareto efficiency/optimality

Vilfredo
Pareto
1848
1923
[Wikipedia]

Subset of (performance, power) tuples in state space


Cannot increase performance without also using more power
Cannot reduce power without also decreasing performance
States that achieve goals will lie on the frontier
Minimum E, ED, ED2,
Minimum power with/without a performance target
Maximum performance with/without a power cap

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

27

EP on Reconfigurable Systems

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

28

EP on Reconfigurable Systems

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

29

Reconfigurable System Efficiency

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

30

Reconfigurable System Efficiency

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

31

Reconfigurable System Efficiency

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

32

Limitations of conventional views


Systems may achieve better than ideal efficiency
Better than the best!!
Neither EP nor Dynamic EP is ideal
Conventional models based on fixed-resource

assumptions
Modern systems are highly reconfigurable

New need ideals and new metrics


Quantify energy waste w.r.t ideal
Quantify factors that contribute to waste
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

33

Outline
Background & Motivation
New ideals
New metrics
Management
Summary

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

34

Conventional Ideals
Design Ideal
For system designers
Help design energy efficient systems
Examples: EP
Helped drive down idle power in current systems

Operational Ideal
Characterizes maximum operating efficiency
Help system operators and OS schedulers?
Example: Dynamic EP
Quantifies divergence from ideal linear behavior

Need new ideals for reconfigurable systems


2/4/16

UNIVERSITY OF WISCONSIN-MADISON

35

New Ideals
Design Ideal
EOP Energy Optimal Proportional
Determined by max
Current systems energy optimal configuration
Best we can do irrespective of of offered load

Operational Ideal
Dynamic EO Dyanamic Energy Optimal
Determined by the Pareto frontier
Realizable ideal for the current system
Best we can do for the offered load

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

36

Conventional vs New Ideals

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

37

New Design Ideal

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

38

New Operational Ideal

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

39

Conventional vs New Ideals


Conventional

New

Design

EP

EOP

Operational

Dynamic EP

Dynamic EO

Dynamic EP EP EOP
Dynamic EO EOP
non-Sub-Linear Dynamic EP Dynamic EO

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

40

Optimality vs Proportionality
Optimality at every load Proportionality
EOP is always proportional
Uses Emin at every load

Emin = Energy needed by EOP


Proportionality Optimality
EP may not be optimal
Uses Emin energy at every load

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

41

Goal Dependence
Minimize E
EOP uses power linearly proportional to load
Constant regardless of load
Minimize ED
EDOP uses power quadratically proportional to load
Minimize ED2
ED2OP uses power cubically proportional to load

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

42

Outline
Background & Motivation
New ideals
New metrics
Management
Summary

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

43

PUE: Power Usage Effectiveness


Building Load
Power
Total
Facility
Energy

IT Equipment
Energy
Cooling

PDUs
Switchgears
Generators
Chillers
UPS
Pumps

CRAHs
Misc
CRACs
DX
Lighting

Total Facility Energy


PUE =
IT Equipment Energy 1
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

CPUE
?

IT Load
Servers
Storage
Networks
Monitors
Workstations
Laptops

[The Green Grid]

44

PUE: Power Usage Effectiveness


CPUE

Total Facility Energy


PUE =
IT Equipment Energy 1
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

[The Green Grid]

45

CPUE: Computational PUE


Goal: To quantify waste in computational energy
Inspired by PUE
Informally:

Actual IT server energy


CPUE =
EOP server energy 1

Depends upon configuration c and load l > 0

Actual server energy with c at l E(c,l)


CPUE(c,l) =
= E
1
EOP server energy
min

E(c,l) = CPUE(c,l) Emin


CPUE(c,l) > 1 wastes energy
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

46

Decomposing CPUE
For configuration c and load l > 0

Actual server energy with c at l


CPUE(c,l) =
EOP energy
E(c,l)
= E
min
& max
:
=
& (c,l)
:
Iron Law

&:max

&:Pareto l

&:Pareto l

& c,l
:

= LUE() RUE(c,l)
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

47

Iron Law of Performance


Iron Law of Performance:


Factor performance to focus designers attention
I/P: Instruction set architectures, compilers, algorithms
CPI: Pipelines, caches, predictors, out-of-order execution
T/C: Technology, circuits, pipelines

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

48

Iron Law of Energy Efficiency


Iron Law of Energy Efficiency:

E(c,l) = LUE(l) RUE(c,l) Emin


Factor energy efficiency to focus designers attention
LUE: Load Usage Effectiveness
Operate server at efficient load level

RUE: Resource Usage Effectiveness


Configure server to run efficiently at a given load level
Emin: Optimal energy for a given computation on this server

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

49

LUE: Load Usage Effectiveness

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

50

LUE: Load Usage Effectiveness

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

51

RUE: Resource Usage Effectiveness

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

52

RUE: Resource Usage Effectiveness

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

53

Energy Waste: Root Cause Analysis


Non-optimal loads
LUE(l) > 1 energy waste
There exists another load that uses less energy

LUE(l) = 1 system operating with &max


Only at points where Dynamic EO and EOP lines meet

Non-optimal configurations
RUE(c,l) > 1 energy waste
There exists another configuration that uses less energy but can

serve the same load


RUE(c,l) = 1 c is at Dynamic EO

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

54

Outline
Background & Motivation
New ideals
New metrics
Management
Summary

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

55

Load Management
Goal: Choose l such that LUE(l) is reduced
Select subset of Dynamic EO close to EOP
Global: Inter-server
Load distribution among servers
Difficult for stateful, time-sensitive services

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

56

Load Management
Optimization problem
Assume n servers, ith servers Dynamic EO is described by set
{(xi,1, yi,1), , (xi, j, yi, j), }
Let L be the total load to be served

Minimize
Such that

Pwr = WXYZ V , ,
L WXYZ V , ,
Ii, j {0,1}
V , 1

for all i, j
for all i

Load for server i is V , ,


2/4/16

UNIVERSITY OF WISCONSIN-MADISON

57

Configuration Management
Goal: Choose c such that RUE(c,l) is reduced
Operate system at/near Dynamic EO
Local: Intra-server
Existing Linux governors not adequate
Limited knobs (DVFS)
Ondemand governor peak performing configuration
Powersave governor (lowest frequency configuration)
Limits performance
Operates in high LUE region

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

58

Reactive Governor
Knobs: core frequency and cache prefetch enable
R(t): parametrized by length of interval = t ms
Track Active Cycles for last interval
Select highest frequency such that all cycles are expected to be
active
Exponential Ramp-up
Inspired by Ethernet backoff
If selected frequency current frequency
Increase selected frequency by step
Double step

Prefetch control
Profile with OFF and ON for (10 + 10) ms within every interval
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

59

Reactive Governor
PF

PF

perf1 >
perf2?
Y
PF

perf1 perf2

#Active Cycles
in last t ms?
Calculate target
freq (tfr)
tfrf?

ftfr

tfr+=step Y
step*=2

T=s T=s+10 T=s+20

N
Init step

T=s+t-20

f = current frequency
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

60

Reactive Governor

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

61

Reactive Governor: RUE

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

62

Reactive Governor

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

63

Reactive Governor: RUE

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

64

SLA-Aware Governors
Disconnect between user needs and OS governors
SLAs
Maximize energy efficiency
Subject to max. response time

Maximize performance within a power cap


RAPL mechanism only seems to deal with DVFS
Other knobs?
Minimize power for a performance target

Intel Skylake Speed Shift


Hardware P states (DVFS)
Other knobs?

OS specifies performance hints


2/4/16

UNIVERSITY OF WISCONSIN-MADISON

65

Challenges
Available knobs?
Hardware has many knobs
But few exposed to operating system
Efficient construction of Pareto frontier (Dynamic EO)
Transition overhead
Speed Shift: 1 msec
Caches? Interconnects?
Measurement interval
RAPL: few (>1) ms
Wall power: 0.1 1 sec
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

66

RAPL Power System Power


Advantage: shorter reaction times

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

67

Summary
Conventional ideal models no longer adequate
EP does not imply Emin
New ideals
EOP as a primary design goal
Dynamic EO as a primary operational goal
New metrics: CPUE, LUE, RUE
Quantify & attribute server energy waste
New SLA-aware governors needed
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

68

Acknowledgments
Rathijit Sen
Ph.D. Candidate

Funding
NSF grant CCF-1218323
NSF grant CNS-1302260
Financial interest in AMD, Google
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

69

Backup

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

70

References
NRDC, Anthesis: http://www.nrdc.org/energy/files/data-

center-efficiency-assessment-IP.pdf
Dimension Data:

https://www.dimensiondata.com/Global/Downloadable%20D
ocuments/The%20Relationship%20Between%20Data%20Cen
tre%20Strategy%20and%20Energy%20Efficiency%20Whitepa
per.pdf
The Green Grid:

http://www.thegreengrid.org/~/media/WhitePapers/WP49PUE%20A%20Comprehensive%20Examination%20of%20the
%20Metric_v6.pdf
2/4/16

UNIVERSITY OF WISCONSIN-MADISON

71

References
PUE numbers
Allied Control: http://multimedia.3m.com/mws/media/1127920O/2phase-immersion-coolinga-revolution-in-data-center-efficiency.pdf
Facebook: https://www.facebook.com/PrinevilleDataCenter/app/399
244020173259/
Google: http://www.google.com/about/datacenters/efficiency/intern
al/index.html#measuring-efficiency
Green IT Cube: http://www.heise.de/newsticker/meldung/Green-ITCube-Hocheffizientes-Supercomputer-Domizil-eingeweiht3082605.html
Microsoft: http://download.microsoft.com/download/8/2/9/8297F7
C7-AE81-4E99-B1DBD65A01F7A8EF/Microsoft_Cloud_Infrastructure_Datacenter_and_
Network_Fact_Sheet.pdf
OVH: https://www.ovh.com/ca/en/about-us/green-it.xml
Average: http://www.datacenterknowledge.com/archives/2014/06/0
2/survey-industry-average-data-center-pue-stays-nearly-flat-fouryears/

2/4/16

UNIVERSITY OF WISCONSIN-MADISON

72

Оценить