Ricardo E.

Gonzalez, Jeffrey Kuskin, and Ron Ho

Mark Horowitz
and His Impact on
Computer Architecture
His work in energy efficiency
and parallel multiprocessing

I
n the early 1990s, computer archi-
tecture research, both academic and
industrial, was flush with big problems
worthy of attack. Designers were trans-
lating workloads from minicomputers to
the microprocessor, and so many classic performance
Digital Object Identifier 10.1109/MSSC.2016.2580259
Date of publication: 2 September 2016
problems were still paramount: systems designers were
pushing on instruction-level parallelism using reorder
buffers and speculation, multilevel branch prediction,
and of course multiscalar machines. They were also
working to tear down the memory wall, leveraging inclu-
sion as a key cache design concept, and extending caches
to hold both eject and inject data.
Among the many microprocessor architecture problem
areas attracting attention were two in particular: energy

IEEE SOLID-STATE CIRCUITS MAGAZINE

1943-0582/162016IEEE su m m e r 2 0 16 31
efficiency and parallel multipro-
cessing. Although interrelated, they
were distinctly separate windows
into the design of large-scale com-
puters. Energy efficiency was a new
focus for many architects and laden
with semantics that often confused
power with work. More importantly,
designers were pushing full throttle
on the gigahertz race and needed to
Mark embodied one of the most under-
appreciated virtues of a faculty advisor:
he pushed students hard to finish their
doctoral degrees.
understand what limits were rap-
idly approaching. By contrast, paral-
lel machines really implied scalable
coherency, and the big question was
how to enable that in a productive
and efficient manner. In these two
areas, Prof. Mark Horowitz, along
with his graduate students at Stan-
ford who focused on architecture,
built groundbreaking systems, dem-
onstrated new design paradigms, and
articulated a number of fundamental
observations that laid the ground
work for much of computer architec-
ture to come.
2.0
1.8
1.6
Supply Voltage (V)
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0.0 0.1
FIGURE 1: Counters of a constant energy-delay-product for CMOS circuits under variability
in voltage, threshold, and temperature (from [1]).
32 su m m e r 2 0 16 IEEE SOLID-STATE CIRCUITS MAGAZINE
Energy Efficiency
Many architects up to the late 1980s
relied on the concept of millions
of instructions per second (MIPS)
per watt to articulate notions of
efficiency. But using this ratio of
instruction throughput to power is
nothing more than the inverse of
power-delay product and, hence,
simply the inverse of energy.
In the era of Intels Pentium proces-
sor, as CMOS gained dominance over
bipolar devices, the problem with
optimizing systems for MIPS per watt
(energy) was thrown into sharp relief.
That is because CMOS circuits can
gracefully handle a dialed-down sup-
ply voltage, unlike bipolar systems
that lose levels of cascaded logic with
reduced supplies. And then, because
power depends quadratically on sup-
ply, the solution to a power-delay
product optimization is to sim-
ply turn down the voltage. Delay
degrades approximately linearly with
= 1.3
0.5
0.8
0.2 0.3 0.4 0.5 0.6
Threshold Voltage (V)
supply, but power falls even faster!
Needless to say, this solution pro-
duces very slow, and hence relatively
uninteresting, systems.
In a pair of early papers, Mark and
Ricardo Gonzalez [1], [2] focused on
the simple semantics of measuring
energy and performance and arti
culated the benefits of optimizing
CMOS circuits for energy-delay prod-
uct. They first focused on a generic
model of CMOS circuits; later, with
the commercial adoption of back-
biasing allowing designers to easily
modify threshold voltage, they also
looked at trading off leakage and
switching power to optimize efficien-
cy. See, for example, Figure 1, which
indicates what choices of thresh-
old voltage and supply voltage offer
the best energy efficiency, assuming
variability in environmental condi-
tions The interesting observation
from this work arose from sensitiv-
ity studies around manufacturing
variability. For designs operating close
to the minimum energy point, these
transistor variations would have
an enormous effect on performance,
making the energy-delay product
of that operating point much worse
than it would appear if variations
were not considered. This work was
based on looking at the delay of
simple inverterswith the rationale
that not only do the power and per-
formance of most CMOS gates scale
similarly to inverters but also the sad
but true fact that most of the gates in
a chip actually are inverters.
In later work on efficient designs,
Mark and his students realized that
using a single metric like energy-
delay product might still lead to sub-
optimal designs, since it assumes
that the designers goal is to evenly
trade off between energy and delay.
Clearly not all designs share this as-
sumption. Papers by Dinesh Patil and
Sameh Galal leveraged the concepts
of Pareto optimality curves in the
broad design space of energy-per-
operation and performance. This al-
lowed them to articulate families of
machines that are highly energy effi-
cient for single-thread performance,
as well as machines focused on
energy-per-operation and area effi-
ciency (mm2-per-operation-per-second)
for throughput performance [3], [4].
Figure 2 illustrates the tradeoffs be-
tween area and energy and how maxi-
mally efficient systems could span
the solution space along the blue
curve, and how real-world constraints
like power density can motivate a par-
ticular solution as optimal.
From his work on modeling effi-
cient processors, it became evident
that caches and local memories play
an enormous role in making the sys-
tem efficient. Fetching a bit from an
off-chip DRAM utterly dwarfs the en-
ergy cost of any on-chip computation
or transfer, so any workload relying
on sustained DRAM usage has al-
ready lost the energetics war. On-chip
storage is thus a key component of
system efficiency, and so Mark looked
at building large and efficient SRAMs.
In a series of papers, Bhadaraj Am-
rutur and Ken Mai developed simple
techniques to leverage the regularity
of memory arrays to cut energy
dissipation per access, for example,
by switching high-capacitive lines at
a reduced voltage but doing so on
both gate and source nodes of an ac-
cess transistor. In later work, Mai ex-
plored the generalization of efficient
memory arrayshow one would
build a flexible array fabric that
could be used either as a traditional
cache (with tags and read-modify-
write mechanics for least-recently
used markers), as a scratch pad, as
storage for a gather-scatter engine,
or as an atomic/synchronization
buffer. This work, to minimize the
overhead of specialized hardware
from generalized building blocks,
has proven remarkably prescient
with todays emergent work on
work load-specific lang uages, al-
gorithms, and hardware [5][7] and
provides the hardware bridge to the
realization of those application-tar-
geted systems.
Interconnects, both off and on
chip, act as another key consumer of
energy. Marks impact on the field of
serial links will be explored in other
IEEE SOLID-STATE CIRCUITS MAGAZINE su m m e r 2 0 16 33
articles in this issue of IEEE Solid-
State Circuits Magazine, but Guyeon
Wei focused on optimizing off-chip
interconnect energy using adaptive
supply regulation [8]. Of course, on-
chip interconnect, especially in a
future of on-chip mesh networks and
tiled multiprocessors, was not to be
overlooked, and Ron Ho wrote papers
enumerating wire scaling challenges
and low-energy usage models [9],
In a pair of early papers, Mark and Ricardo
Gonzalez focused on the simple semantics
of measuring energy and performance and
articulated the benefits of optimizing CMOS
circuits for energy-delay product.
[10]. One of the motivating obser-
vations Mark made was that, until
then, architects and logic designers
had the luxury of ignoring wires:
they were relatively fast compared
to logic gates, and hardware abstrac-
tion languages like Verilog inferred
wires by simply reusing variables
across lines of code. But this was
changing. Wires were scaling in the
reverse direction as gates, and to
understand the system implications,
architects and architecture students
needed a how to use wires guide. So
Mark produced one [9].
Along with memories and inter-
connect, compute circuits form the
0.05
0.04
P (W/GFlops)
0.03
0.02
0.01
0
0 0.02
FIGURE 2: Energy and throughput Pareto tradeoffs under different constraints (from [4]).
third leg of the microprocessor archi-
tecture tripod, and they can be viewed
from the lens of energy efficiency as
well. Patil and Galal employed convex
methods to optimize integer adders
and floating point units for energy
delay. Some interesting observations
arose, for example, in the case of
adders; many already understood that
different parallel-prefix adder struc-
tures could minimize two of either
logic stages, wire tracks, or fanout.
However, Patil pointed out that the
energy-efficient choice was, surpris-
ingly, to minimize logic stages and
tracks and give up on fanout for the
simple reason that logical fanout
and electrical fanout are not cre-
ated equal. A gate that must drive six
others can actually drive one critical
gate and five super-small noncritical
side loads.
Certainly a higher architectural
perspective helps to ties together
the individual components. Gonzalez,
and later Omid Azizi, focused on these
system-level explorations [11]. The
latter work, in particular, ambitiously
2
/cm
W
50 2
x
=
W/cm
D Ma = 30
ax
x
/ AM
P Ma
Optimal Design
0.04 0.06 0.08 0.10 0.12 0.14
A (mm2/GFlops)
 FLASH was an enormous research success,
spanning over five years of development
and graduating more than 20 Ph.D. and M.S.
students, and Mark was the primary faculty
member behind it.
modeled an entire microprocessor to
extract incremental cost and benefit
from a host of architectural features,
again leveraging economic concepts
of Pareto optimality. Figure 3 shows
a broad survey of commercial pro-
cessors, demonstrating empirically
a boundary of energy efficiency that
has emerged over decades of indus-
try practice.
Parallelism and Scalable
Shared Memory
Multiprocessing and the attempt to
bring massive parallelism to computer
architecture had its roots in the mini-
computer era. By the early 1990s, many
had turned toward cache coherency
as a foundational problem to solve.
Designers envisioned distributing me
mory among each of many so-called
node cards, giving each processor fast
local memory, but the primary debate in
the computer architecture community
was how to keep all those memories

Six observations on Mark Horowitz

Observation 1
Anyone who volunteers for IEEE or works on a conference committee
will recognize the syndrome. You find yourself in a room full of people
you dont know very well, but you know their names and their reputa-
tions, and you think I sure hope I dont make a fool of myself. Then
the saying better be silent and be thought a fool, than to speak up and
remove all doubt pops into your head. The discussion begins, and
you spend half your time thinking what the heck did they just say?
and the other half thinking wait, that cant be right. And then Mark
Horowitz speaks up, cuts right to the heart of whats being discussed,
and in a reasonable, nonoffensive way, gets two key points across:
1) youre all on the wrong track and 2) heres the right one. Its a beauti-
ful thing to watch.
Observation 2
Mark doesnt disdain philosophical and abstract discussions per se. In-
stead, he deftly substitutes a quantitative argument that illuminates the
actual issue and effectively indirectly quashes the angels on a pin ar-
gument that had been holding sway. In so doing, he single-handedly
lowers the frustration level in the room, even on the part of the people
who were so intently estimating angel populations.
topics not in that set. By contrast, if Mark asserts something, you had
better have a really, really good reason for disputing it.
Observation 4
Theres a related issue, and again, Mark exemplifies the right way to do
it: hes a good listener. When others are making legitimate points, he
listens intently, and then he often does a remarkable thing (remarkable
for how seldom it happens): he asks follow-up questions intended to
help the other person with their exposition and, also presumably, to help
Mark clearly understand what theyre proposing. Too often, tech folks in
a similar position are simply waiting for whoever has the floor to shut up
so that they can start talking. We should all be more like Mark.
Observation 5
Mark is world class at knowing what level of abstraction is the right one for
helping to resolve or clarify whatever is under discussion. Sometimes that
level is so low that atomic physics and thermodynamics are the entrance
fee; other times the big picture is required so that the details dont over-
whelm. No matter what level is the right one, Mark will think carefully, con-
sider what others are saying, and then reliably offer up some trenchant ob-
servation that is universally on target, interesting, insightful, and influential.

Observation 3
Marks not afraid to say the words that, in certain tech circles, are almost
never uttered: I dont know. Nobody can possibly know everything,
but the percentage of people willing to fake it is awfully high sometimes.
The problem seems not quite so chronic these days, when all the silent
observers (see Observation 1) can instantly Google-check any particu-
larly sketchy assertions made by inexplicably confident patriarchs of the
field. (I cant think of any cases where women pulled this stunt. Its usu-
ally the men.) Not so long ago, however, meetings were routinely de-
railed by people who actually were bona fide experts in a set of certain
things but who were equally willingand equally likelyto opine on
Digital Object Identifier 10.1109/MSSC.2016.2580260
Date of publication: 2 September 2016
Observation 6
Finally, you know that guy in the ads, where they say hes the most
interesting man in the world? I really dislike that guy and that ad. But in
thinking about why I hate the ad, I realized that I have come to really
value people who are, in fact, interesting. They notice things about the
world, and then they think deeply about what their observations might
mean about the way the physical world is and the way it works. I think
Mark is one of these people: hes interesting!
Bob Colwell
About the Author
Bob Colwell most recently served as director of the Microsystems Tech-
nology Office at DARPA. He was an Intel Fellow and the chief IA-32 ar-
chitect at Intel for the Pentium Pro, II, III, and 4 microprocessors, retiring
in 2000. He received the ACM Eckert-Mauchly Award in 2005.

34 su m m e r 2 0 16 IEEE SOLID-STATE CIRCUITS MAGAZINE

coherentand whether a scalable
cache-coherency system could be rea-
sonably implemented in hardware and
run fast enough. Some argued it could
be done and that the system could
maintain a coherent shared memory
image without the programmers help.
Others pushed for simpler distributed
machines that could only share coher-
ency information through explicit mes-
sage passing.
At Stanford, Mark worked on an early
machine, the DASH multiprocessor,
that implemented a ground-break-
ing 64-node machine with distrib-
uted cache-coherent shared memory
and could support several scientific
applications in the SPLASH suite of
parallel applications [12]. Two of the
faculty sponsors of DASH, Mark and
Stanford President John Hennessy,
not only guided the hardware devel-
opment but also helped to facilitate
an early and highly successful tech-
nology transfer to industry. The Sili-
con Graphics Origin 2000 was one of
the first commercial hardware-based
cache-coherent systems, and it was
architected and built by many of the
same students who had built DASH
and were later hired by SGI. This was
an example of the general principle
articulated by computer pioneer Bert
Sutherland that technology transfer
is a contact sport.
Almost immediately after com-
pleting DASH, Mark began the de
vel opment of the next Stanford
parallel system, called FLASH, to
explore questions around the paral-
lel programming model and how it
related to distributed shared mem-
ory versus message passing and the
data structures and implementa-
tions on which those programming
models are built [13]. One of Marks
observations motivating FLASH was
that ASIC development had suffi-
ciently advanced to the point that
a reasonably complex system could
be built by a small team of motivated
graduate students. That opened the
door to build a programmable imple-
mentation of a so-called node con-
troller (the engine providing the
cache coherency model primitives),
IEEE SOLID-STATE CIRCUITS MAGAZINE su m m e r 2 0 16 35
Most importantly, when needed, [Mark] gave
critical autonomy to the graduate students
who made up the design team.
instead of the hardwired implementa-
tion that DASH used.
By creating a custom and flexible
substrate for multiprocessing, FLASH
could offer an integrated and stream-
lined set of hardware primitives for
both global cache coherence and also
user-level message passing. It was, in
effect, the ideal research vehicle to test
and compare both type of systems.
FLASH was an enormous research
success, spanning over five years of
development and graduating more
than 20 Ph.D. and M.S. students, and
Mark was the primary faculty member
behind it. He helped to guide major
architectural and design decisions of
the node controller (nicknamed the
MAGIC chip), and he steered the team
not only in creating a useful simulation
framework to estimate its performance
but also in using those results to cap-
ture the key research questions FLASH
needed to answer. Of course, with any
group of bright-eyed graduate students,
100
(Nomalized for technology)
Energy Operation
10
1
0.00
Intel 80386
Intel Pentium III
Intel Core 2
Alpha 21064
HP PA
AMD K7
AMD Phenom
FIGURE 3: A survey of microprocessors on a Pareto curve for the energy-per-operation and
performance (from [11]).
there is always the danger of over-scop-
ing the problem, and Mark played a
critical role in containing the untrained
appetites of the team to ensure that the
machine would be buildable. But per-
haps most importantly, when needed,
he gave critical autonomy to the gradu-
ate students who made up the design
team. Led primarily by Jeff Kuskin
and Dave Ofelt, along with a talented
team of other doctoral candidates, the
team exemplified learning by doing, all
under Marks mentoring.
As with any large-scale system,
there were efforts down in the silicon,
with Mark teaching the team to build a
custom six-ported memory whose area
and frequency was critical to the node
controllers performance. He helped
tackle computer-aided design (CAD)
issues in finding out what new tools were
being developed, either in the commer-
cial space or internal to companies,
and how to get access to those tools
for his students. He worked to get large
Energy-Efficient Frontier
0.01 0.10 1.00
Performance (Normalized for Technology)
Intel 80486 Intel Pentium Intel Pentium II
Intel Pentium IV Intel Itanium Intel Itanium D
Intel Xenon Intel Atom Intel Core i7
Alpha 21164 Alpha 21264 MIPS
Power PC IBM-Power AMD K6
AMD Turion AMD Thlon AMD Operation
Sun Super Sparc Sun Ultra Sparc
 Computer architecture, much like the rest
of the silicon revolution, has undergone
enormous transformations over the past few
decades, and hand-held compute systems we
casually store in our pockets today are
profoundly more powerful than the
mainframes of the past.
ASICs fabricated with the sponsorship
of companies like Silicon Graphics, Inc.
and LSI and then taught the team how to
package, test, and sort the chips that
resulted. He was equally at home at a dry
erase board drawing stick figures of lay-
out to floor plan critical circuits, wield-
ing a soldering iron to swap out a bad
connector, showing the students proper
oscilloscope techniques (hint: never use
the auto-scale button around Mark),
or grabbing a screwdriver to assemble
a chassis.
Finally, Mark embodied one of the
most under-appreciated virtues of a
faculty advisor: he pushed students
hard to finish their doctoral degrees. He
was an exacting mentor, but he worked
to ensure that students were driving
toward an actual conclusion to their
Ph.D. tenures. If he was one of your
advisors, you knew the weekly grill-
ing, painful as it might be at times, was
bringing you closer to completion.
Final Thoughts
Computer architecture, much like the
rest of the silicon revolution, has un-
dergone enormous transformations
over the past few decades, and the hand-
held computer systems we casually
store in our pockets today are profound-
ly more powerful than the mainframes
of the past. One of the hallmarks of this
progress has been a virtuous feedback
loop in which advances in all aspects of
computer technology become fun-
damental enablers for building new-
er, faster, and more powerful machines.
Mark and his students have been an in-
tegral part of this cycle, making signifi-
cant and sometimes paradigm-shifting
contributions across the space. Here we
36 s u m m e r 2 0 16 IEEE SOLID-STATE CIRCUITS MAGAZINE
discussed advances in energy efficien-
cy and parallel distributed memory, and
we simply lacked the space to continue
the narrative into CAD tools, circuits,
microarchitectures, compilers, and
so on. But partly due to these contribu-
tions, each generation of computers has
been just powerful enoughin compute
performance, or storage, or interconnect
bandwidth, or richness of displayto
build the next generation of comput-
ers. This feedback loop has allowed
the engineering community to repeat-
edly bootstrap design after design and,
thus, put incredibly powerful systems
in large data farms, in our cars, in our
pockets, and inextricably woven into
the fabric of everyday life.
References
[1] R. Gonzalez and M. Horowitz, Energy
dissipation in general purpose micropro-
cessors, IEEE J. Solid-State Circuits, vol.
31, pp. 9, Sept. 1996.
[2] R. Gonzalez and M. Horowitz, Supply
and threshold voltage scaling for low-
power CMOS, IEEE J. Solid-State Circuits,
vol. 32, pp. 8, Aug. 1997.
[3] D. Patil, O. Azizi, R. Anantharaman, R. Ho,
and M. Horowitz, Robust energy-effi-
cient adder topologies, in Proc. IEEE
Symp. Computer Arithmetic, 2007.
[4] S. Galal and M. Horowitz, Energy-efficient
floating-point unit design, IEEE Trans.
Computers, vol. 60, pp. 7, June 2010.
[5] B. Amrutur and M. Horowitz, Speed and
power scaling of SRAMs, IEEE J. Solid-
State Circuits, vol. 35, pp. 2, Feb. 2000.
[6] K. Mai, T. Mori, B. Amrutur, R. Ho, and M.
Horowitz, Low-power SRAM design us-
ing half-swing pulse-mode techniques,
IEEE J. Solid-State Circuits, vol. 33, pp. 11,
Nov. 1998.
[7] K. Mai, T. Paaske, N. Jayasena, R. Ho, W. Dally,
and M. Horowitz, Smart memories: A modu-
lar reconfigurable architecture, in Proc. Int.
Symp. Computer Architecture, 2000.
[8] G. Wei and M. Horowitz, A fully digital,
energy-efficient adaptive power supply
regulator, IEEE J. Solid-State Circuits, vol.
34, pp. 4, Apr. 1999.
[9] R. Ho, K. Mai, and M. Horowitz, The fu-
ture of wires, Proc. IEEE, vol. 89, pp. 4,
Apr. 2001.
[10] R. Ho, K. Mai, and M. Horowitz, Efficient
on-chip global interconnects, in Proc.
IEEE Symp. VLSI Circuits, 2003.
[11] O. Azizi, Design and optimization of
processors for energy efficiency: A joint
architecture-circuit approach, Ph.D.
dissertation, Stanford University, CA,
2010.
[12] D. Lenoski, J. Laudon, K. Gharachorloo, W.
Weber, A. Gupta, J. Hennessy, M. Horow-
itz, and M. Lam, The Stanford DASH mul-
tiprocessor, IEEE Computer, vol. 25, pp. 3,
1992.
[13] J. Kuskin, D. Ofelt, M. Heinrich, J. Hein-
lein, R. Simoni, K. Gharachorloo, J. Chap-
in, D. Nakahira, J. Baxter, M. Horowitz, A.
Gupta, M. Rosenblum, and J. Hennessy,
The Stanford FLASH multiprocessor, in
Proc. Int. Symp. Computer Architecture,
1994.
About the Authors
Ricardo E. Gonzalez received the B.S.,
M.S., and Ph.D. degrees in electrical
engineering from Stanford University,
California. He was a member of found-
ing teams at Tensilica and Stretch,
where he led the development of con-
figurable and extensible processors.
He has also worked at Intel, VMware,
and Pure Storage. In the fall, he plans
to pursue an M.S. degree in ecosystems
and climate change from Imperial Col-
lege London.
Jeffrey Kuskin received an un
dergraduate degree from Dartmouth
College and M.S. and Ph.D. degrees in
electrical engineering from Stanford Uni-
versity, California. From 1997 to 2000, he
designed large-scale distributed shared
memory multiprocessors at Silicon
Graphics. From 2000 to 2004, he devel-
oped wireless networking chip sets at
Atheros Communications. Since 2004,
he has been at D.E. Shaw Research,
designing s pecial-purpose hardware
to accelerate molecular dynamics sim-
ulations of proteins and other biologi-
cal macromolecules.
Ron Ho (ronho@ieee.org) received
his undergraduate, masters, and
doctoral degrees from Stanford Uni-
versity, California. He is currently
director of Interconnect IP at Altera,
now part of Intel Corporation. From
2003 to 2014, he was with Sun Micro-
systems (later Oracle Corporation),
and from 1993 to 2003 he was with
Intel Corporation.