Instrumentation and

Measurement
CSci 599 Class Presentation
Shreyans Mehta

Abstract

Why Instrumentation and Measurement ?

Instrumentation Techniques
Resources
Data Analysis
Case Study: Paradyn
Guiding Principles
System Overview
W3 Search Model

Why Instrumentation and Measurement ?

Gathering data to improve the next
execution of the program.
Guiding scheduling decisions
Adapting to computations while in
execution

Instrumentation Techniques
Program Instrumentation Techniques
Manual : Programmer inserted directives
Automatic : No direct user involvement
Binary Rewriting
Dynamic Instrumentation

Processor Instrumentation Techniques

Information includes timers, memory system
performance, processor usage, etc.
Available mostly through special registers or memory
mapped location.
Example : Pentium Pro provides performance data through
MSRs. These registers include 64 bit cycle clock and counts of
memory read /write, L1 cache misses, pipeline flushes, etc.

Hardware assisted trace generation.

 Operating System Instrumentation Techniques

Information includes behavior of virtual memory, file
system, file cache etc.
Instrumentation in the form of APIs for applications to
access these variables.

Network Instrumentation Techniques

Ways of measuring
Passive
Example: RMON protocol defines SNMP MIB variables to
report traffic statistics over hubs and switches.

Active
Example: Ping, NWS in grid style computing.

Data Storage Representation

Scalars
Counters
Times

Traces
Vector series

Resources
Software Abstractions

Program Components
Code in Executions
Synchronization Objects
Other Software Abstractions

Hardware Abstractions
Network Abstractions

Data Analysis
Quantitative Performance
Automating Performance Diagnosis
Perturbation Analysis

The Paradyn Parallel

Performance Measurement Tools
Case Study

Guiding Principles and

Characteristics

Scalability
Automate the search for performance problems
Provide well-defined data abstractions
Support heterogeneous environments
Support high level parallel languages
Open interfaces for visualization and new data
sources
Streamlined use

System Overview
Basic Abstractions
Metric-focus grid
Time Histograms

Components of the System

Main Paradyn Process

Performance Consultant
Visualization Manager
Data Manager
User Interface Manager

Paradyn daemons
External Visualization Processes.

Table Visualization

Histogram Visualization

Paradyn

Tabular Summary
CPU

3.0 4.0

Messages 117 81

Visualization
Manager
Visi Thread

Visi Thread

User Interface
Manager

Performance
Consultant

Data Manager

Metric Manager
Instrumentation
Manager

Application

Processes

Metric Manager

Paradyn Daemon(s)

Instrumentation
Manager

Application

Processes

Dynamic Instrumentation
Dynamic Instrumentation Interface
Metric Manager
Instrumentation Manager

Points, Primitives and Predicates

addCounter(fooFlg, 1)
addCounter(fooFlg, 1)
if (fooFlg)
startTimer(msgTme, ProcTime)
if (fooFlg)
stopTimer(msgTme)

Foo()
{
.
.
}
SendMsg( dest, ptr, cnt, size)
{
.
.
}

 Instrumentation generation
Base Trampolines
Mini-Trampolines

Data Collection
Internal Uses of Dynamic Instrumentation
Resource Discovery
Collection of dynamic mapping information for
HLL.

The W Search Model and the

Performance Consultant
3

Why ? Where ? When ?

The Why Axis
Why is the application performing poorly ?
Potential performance problems are represented as hypotheses
and tests.
Hypotheses represent activities universal to all parallel
computations.
Hypotheses can be refined into more refined hypotheses using a
search hierarchy.
Tests are Boolean functions that evaluate the validity of a
hypotheses.
Tests are expressed in terms of a threshold and metrics
calculated by the Instrumentation Manager.

A sample why axis with several hypotheses

TopLevelHypotheses

SyncBottleNeck
FrequentSyncOperations

HighSyncBlockingTime

HighSyncHoldingTime

HighSyncContention

 The Where Axis

Where is the performance problem ?
Pinpoints the problem specific to program components.
Each hierarchy in where axis has multiple levels, with the
leaf nodes being the instances of resources used by the
application.
SyncObject

Semaphores

Message

SpinLock

Barier

 The When Axis

When does the problem occur ?
Represents periods of time during which performance
problems can occur.

The Performance Consultant

This module discovers performance problems
by searching the space defined by W3 Search
Model.
Fully automated search but also allows user to
make manual refinements.

Open Visualization Interface

Paradyn provides a simple library and RPC
interface to access performance data in realtime.
Visualization modules (visis) in Paradyn
are external processes that use this library
and interface.
Currently provides visis for timehistograms, bar charts and tables.

Examples of Use

Conclusion
Computational grids are focused on high
performance distributed computing. To
achieve high performance, such systems
need to provide tools that enable the
programmer to realize the potential
performance inherent in such a system.

References
Jeffery K. Hollingsworth and Bart Miller, Instrumentation and
Measurement, Chapter 14 of Grid: The Blueprint for a new
computing infrastructure.
Bart Miller, The Paradyn Parallel Performance Measurement Tools,
http://www.cs.wisc.edu/~paradyn/papers/index.html