11.

3-1

Application Characterization System for

Gaming Consoles
Andrs Hernandez, Cara Yang, Francine Shammami and John Collins
Microsoft Corp., Redmond, WA
Abstract The Application Characterization System (ACS)
allows console manufacturers and testers to see how different
video game titles can stress a game console. With ACS, console
power, temperature data, and thread usage are used in
conjunction to monitor performance, GPU regression,
characterization through video output formats and resolutions
and Gauge R&R.

I. INTRODUCTION
The Application Characterization System (ACS) takes a
dual approach to look at the way in which different game titles
stress specific hardware components of a game console1, using
a combination of hardware and software methods. ACS
accurately captures hardware resource utilization across
platforms and kernels.
Internal teams can use the data to develop test plans to
include game titles that effectively stress the console under a
regular user scenario. Hundreds of titles have shipped, and
load characterization over this catalog is important as the team
plans future cost reductions. All games in the catalog must
function perfectly with each reduction.
ACS data is collected through two primary points: hardware
monitoring and through software counters. They are unrelated
collection processes and can function independently of each
other. The hardware approach directly measures input power
and power consumption of the GPU and CPU. The software
technique pulls information from registers to monitor requests
and log performance counters.
Combined, these sources give a better understanding of the
consoles performance under real-world gameplay. The
following sections describe different studies done using ACS:
Gauge R&R, GPU regression and characterization through
video out formats and resolutions.
II. CHARACTERIZATION THROUGH REPEATABILITY AND
REPRODUCIBILITY

For ACS, repeatability is defined by one tester playing a

single game for several trials under the same conditions,
whereas reproducibility involves several trials of a single game
among different testers with the same conditions as provided
in repeatability. ACS can be used to determine console stress
variation among these two scenarios.
1

Xbox 360, developed by Microsoft Corp.

978-1-4244-4316-1/10/$25.00 2010 IEEE

Figure 1. Mean usage per thread by percentage across varying testers

Figure 1 shows the variability of mean thread usage of a

game run consisting of two trials by three different testers on
the same game with the performance monitoring tool running
on Thread 4. It is apparent that the mean usage of Thread 0 is
constant across the trials regardless of the user. Thread 2 is
user dependent.
Similar comparisons can be done for CPU RMS power,
GPU RMS power, mean pixel vectors submitted, and total
memory access requests.
Gameplay differs from player to player for several reasons.
Players take different routes through games, and this
determines the demands placed on GPU and memory in the
console. For instance, a player who navigates the menu system
will burden the GPU much less than someone who is playing a
realistic looking game.
III. GPU REGRESSION
A correlation can be made between power consumption and
dependence on internal graphics processing. Data used in these
trials includes play and attract modes. In attract mode the
game loops through a preset sequence. Play mode describes
the regular user scenario. Multivariate analysis using all the

GPU counters helped to determine which counters had

stronger correlation coefficients with power. It was also used
to filter those that were strongly correlated with each other.
An initial model for GPU power was developed using the
least squares method for linear regression, assuming a normal
distribution due to the large sample size2, statistical
assumptions were relaxed. Figure 2 shows measured GPU
power data versus GPU predictions from the regression. The
upward slope indicates a good dependency between the two,
but the large scatter of points indicates considerable variance.
The adjusted R-Squared value points to a decent fit,
considering that there are hundreds of counters available and
only a small fraction were collected. Longer game trials where
more counters are used would be ideal for more accurate
analysis.

Figure 3. Pixel vectors submitted during attract mode across varying

resolutions for a select game title

V. CONCLUSION

Figure 2. Actual GPU power data versus predicted data from LS regression.
The correlation between predicted GPU power and actual GPU power is fairly
high, at 0.75.

IV. CHARACTERIZATION THROUGH VIDEO OUTPUT FORMATS

AND RESOLUTIONS

ACS also allows for better understanding of console

behavior at different video output formats and resolutions. A
run with this purpose was performed in 10 minute intervals of
data capture for each resolution and video format
configuration using the same console and game title. The title
was run in attract mode to minimize measurement variance
during game play.
Figure 3 displays pixel vectors submitted. As higher
resolutions require more pixels to be displayed on the screen,
the results can be correlated to show a clear difference in how
the console handles EDTV (480p) and HDTV (720p and
1080p) inside the audio/visual architecture.
In this study, CPU RMS Power was within 1W across the
different video format resolution combinations.
2

Sample size 4,310

ACS can be used in situations where quantitative analysis is

required to determine console stress. The data can be
correlated to show how different user scenarios affect the
hardware
components.
Through
repeatability
and
reproducibility testing, varying testers across multiple trials
does produce different thread usage. Pixels vectors submitted
do change based on video output formats and resolutions. This
data can be used to create test applications. For example, ACS
can provide important feedback for kernel optimization. It can
also aid in the design of new hardware components, as the tool
recognizes games that use the consoles resources the most.
ACKNOWLEDGMENTS
The authors would like to thank mentors Perry Stultz and
Paul Hall for their advice and guidance, as well as JMP for
providing the graphics.
REFERENCES
[1]

D. Montgomery and G. Runge,. Applied Statistics and

Probability for Engineers. New York: John Wiley &
Sons, 2006.
[2] M. George, D. Rowlands, M. Price, and J. Maxey, The
Lean Six Sigma Pocket Toolbook. New York: McGrawHill, 2005.