Вы находитесь на странице: 1из 50

RMF Development Edition

z/OS
Resource Measurement Facility

RMF Technical Overview

March, 6th 2010

2010 IBM Corporation

Resource Measurement Facility

Trademarks
The following are trademarks of the International Business Machines Corporation in the United States and/or other countries.
APPN*
CICS*
DB2*
DB2 Connect
DirMaint
e-business logo*
ECKD
Enterprise Storage Server*
ESCON*
FICON*
GDPS*
Geographically Dispersed Parallel Sysplex

HiperSockets
HyperSwap
IBM*
IBM eServer
IBM e(logo)server*
IBM logo*
IMS
Language Environment*
MQSeries*
Multiprise*
NetView*
On demand business logo

OS/390*
Parallel Sysplex*
PR/SM
Processor Resource/Systems Manager
RACF*
Resource Link
RMF
S/390*
Sysplex Timer*
System z
System z9
TotalStorage*
Virtualization Engine

VM/ESA*
VSE/ESA
VTAM*
WebSphere*
z/Architecture
z/OS*
z/VM*
z/VSE
zSeries*
z9

* Registered trademarks of IBM Corporation

The following are trademarks or registered trademarks of other companies.


Java and all Java-related trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation.
* All other products may be trademarks or registered trademarks of their respective companies.

RMF Technical Overview

2010 IBM Corporation

Resource Measurement Facility

Anything else?

Postprocessor
ZZ Session
Monitor III

Monitor II

Mintime &
Range

Interval &
Cycle Time

% Workflow

RMF Technical Overview

Sysplex
Data Server

Distributed
Data Server

Spreadsheet
Reporter

CIM Data
Provider

% Using / Delay

2010 IBM Corporation

The RMF product consists of several components for all kinds of performance related disciplines. This session introduces
the main RMF components, sheds light on their primary purpose and discusses the most common RMF terms
Even though its impossible to become an RMF expert within one hour: lets get started and lets see how far we get!

Resource Measurement Facility

Agenda





Product Structure
Address Spaces
Controlling the Data Gatherers
Historical Reporting
Postprocessor
Spreadsheet Reporter

Realtime Reporting
Monitor III
Monitor II
WTO Alerts
RMF Performance Monitoring
Data Portal


4

RMF Performance Data APIs

RMF Technical Overview

2010 IBM Corporation

The following topics will be covered in this session:


What are the basic functions and what is the primary purpose of these functions?
What address spaces have to be started? What are their names, underlying procedures and responsibilities?
What are the options which determine how the gatheres work in detail?
What are the functions for historical reporting and after the facts analysis?
What are the functions for realtime reporting and problem determination?
Which API's can be user by components or user written programs to retrieve z/OS performance data from RMF

Resource Measurement Facility

RMF Product Overview


Monitor III Data Portal & RMF PM

RMF Spreadsheet Reporter

Windows, Linux

Windows

RMF Distributed Dataserver (DDS)

RMF Postprocessor
Historical Reporting,
Analysis and Planning

RMF Monitor II and III


Real-Time Reporting,
Problem Determination

RMF Sysplex Data Server and APIs


SMF
SMF
RMF
Data Gatherer

RMF
Monitor I

RMF
Monitor II
background

RMF
Monitor III

VSAM
VSAM

RMF Technical Overview

2010 IBM Corporation

z/OS Resource Measurement Facility (RMF) is an optional priced feature of z/OS. It supports installations in performance
analysis, capacity planning, and problem determination. For these disciplines, different kinds of data collectors are needed:
Monitor I long term data collector for all types of resources and workloads. The SMF data collected by Monitor I is
mostly used for capacity planning and performance analysis
Monitor II snap shot data collector for address space states and resource usage. A subset of Monitor II data is also
displayed by the IBM SDSF product
Monitor III short-term data collector for problem determination, workflow delay monitoring and goal attainment
supervision. This data is also used by the RMF PM Java Client and the RMF Monitor III Data Portal
Data collected by all three gatherers can be saved persistently for later reporting (SMF records or Monitor III VSAM
datasets)
While Monitor II and Monitor III are realtime reporters, the RMF Postprocessor is the historical reporting function for Monitor
I data

Resource Measurement Facility

RMF Address Spaces / Procedures

SYS1.PROCLIB(RMF)

SYS1.PROCLIB(RMFGAT)

SYS1.PROCLIB(GPMSERVE)

RMF
Control
RMF
Monitor III
Gatherer

s RMF

Monitor I
(zz Session)

SY
S
( E 1 .P A
RB
RM RML
F 0 IB
4)

SY
S
(E 1.PA
RB
RM RML
F0 IB
0)

RMF
Distributed
Data
Server

RMF
Monitor III
Batch
Reporting

s GPMSERVE

s RMFM3B

SY
S
(G 1.PA
PM
R
SR ML
V0 IB
0)

f RMF, s III

f RMF,s ZZ

SYS1.PROCLIB(RMFM3B)

RMF Technical Overview

2010 IBM Corporation

In order to make all RMF functions working, a couple of address spaces have to be activated:
RMF is the root address space. It is required for any further activities
The Monitor I gatherer is a subtask in the RMF address space. It is started immediately together with RMF (by default) or
later by means of the modify command
RMFGAT is the Monitor III data gatherer. It cannot be started as standalone address space. Like Monitor I, it can only be
activated with the modify command. Within a sysplex, one instance per system is needed for RMFGAT
The RMF Distributed Data Server is the data source for the RMF Performance Monitoring java client and the Monitor III
Data Portal. Only one instance is needed per sysplex
RMFM3B is needed for the generation of console messages in case a certain threshold is exceeded

Resource Measurement Facility

RMF Control Address Space


keeps Configuration Tables and other Control Information
provides the Command Interface to set of modify Options:
f RMF,f ZZ,MEMBER(99)

maintains the SMF Data Buffer:


s RMF,,,SMFBUF(SPACE(12M),RECTYPE(70:78))
SMF
Exits
IEFU83x

INSTALL

COPY

Sysplex
Data Server

SMFBUF
(DataSpace)

RMF

CALL

SMF

WRITE

Write SMF Record

SMF

SMFWTM

Component

all SMF Record Types can be maintained by the RMF Sysplex Data Server !

RMF Technical Overview

The RMF control address space has 3 major responsibilities:


Builds up configuration tables, reads and remembers the parmlib options
Provides the command interface to set options or to start and stop the gathering functions
Maintains the SMF Buffer of the RMF Sysplex Data Server

2010 IBM Corporation

Resource Measurement Facility

Data Gathering Methods

exact measurement counts


pick up consecutive counters
calculating the difference at the end of an interval

Interval Gathering

eg. CPU seconds, device connect time...

sampling counts
inspect variable counters continuously
building the average at the end of an interval

Cycle Gathering

eg. queue counts, frame counts...

RMF Technical Overview

2010 IBM Corporation

In order to collect all the information that is displayed in the various reports, RMF uses two different techniques of data
gathering
The preferred and cheap method is to retrieve exact measurement counts: this can be timers (e.g. dispatch times) or
accumalitve counts like the Start Subchannel count for a device
The costly cycling technique is needed for counters which are changing rapidly and frequently:
how long is the dispatcher queue?
how many I/O queue elements are chained to a UCB?
what is the actual enqueue situation?

Resource Measurement Facility

Monitor I Data Gathering


1. Measurements
CACHE
CHANNEL
CPU
CRYPTO
DEVICE
ENQ
IOQ
FCD
ESS
PAGESP
PAGING
TRACE
VSTOR
WKLD

SMF 74.5
SMF 73
SMF 70.1
SMF 70.2
SMF 74.1
SMF 77
SMF 78.3
SMF 74.7
SMF 74.8
SMF 75
SMF 71
SMF 76
SMF 78.2
SMF 72.3

SY
S
(E 1.PA
RB
RM RML
F 0 IB
0)

SMF

SMF 74.2
SMF 74.3
SMF 74.4
SMF 74.6

XCF
OMVS
CF
HFS

2. Timing
CYCLE(1000)
NOSTOP
SYNC(SMF)

gathered by
Monitor III

3. Reporting / Recording
RECORD
REPORT(REALTIME)
SYSOUT(A)
4. User Exits
NOEXITS

RMF Technical Overview

2010 IBM Corporation

The directives to setup Monitor I data gathering are kept in SYS1.PARMLIB(ERBRMF00)


There are four different categories of control statements:
Measurements: specifies the kind of resources, which are the candidates for data gathering
The collected data can be saved to SMF records for later reporting
A subset of SMF records is gathered by Monitor III: since the same data is needed for Monitor III reports,
duplicate gathering is avoided
Timing: determines the frequency for interval gathering and cycle gathering
Reporting/Recording: defines the type and the creation time of the output
User Exits: specifies routines which can be executed during session processing to gather or report additional data:
ERBMFIUC (initialization)
ERBMFxxx (cycle gatherer, loaded by ERBMFIUC)
ERBMFDUC (interval gatherer)
ERBMFRUR (report writer)
ERBMFTUR (termination)

Resource Measurement Facility

Monitor III Data Gathering


SY
S
(E 1.PA
RB
RM RML
F0 IB
4)

1. Measurements
IOSUB
CFDETAIL
CACHE
VSAMRLS
OPD
HFSNAME
zFS
SGSPACE
2. Timing
CYCLE(1000)
MINTIME(60)
NOSTOP
SYNC(00)

RMFGAT Address Space


In Storage Buffer
WSTOR(32)

3. Recording
DATASET(ADD(RMF.M3G.&SYSNAME..DS1))
DATASET(ADD(RMF.M3G.&SYSNAME..DS2))
DATASET(START)
DATASET(NOSWITCH)
DATASET(WHOLD(7))

DATASET(WHOLD(7))

4. Buffer
WSTOR(32)
ERBVSDEF vsam_ds VSAMVOL(volser)
DATASET(ADD(vsam_ds))

DATASET(DEL(vsam_ds))
ERBV2S vsam_ds seq_ds

ERBS2V seq_ds vsam_ds

10

RMF Technical Overview

2010 IBM Corporation

The directives to setup Monitor III data gathering are kept in SYS1.PARMLIB(ERBRMF04)
Only a few measurements can be turned ON or OFF. The data collection cannot be controlled on resource granularity:
almost all resources must be monitored in order to get the complete picture of the address spaces which are using or
delayed for certain resources. However, a subset of gathering activities which does not interfere with address space states
can be switched on or off
In order to extend the report range for historical data, Monitor III VSAM Datasets can be used. In this case, the buffersize
which has been specified by the WSTOR option can be reduced to the WHOLD size (since the data which does not fit any
more in the buffer is kept at least in the datasets)
For a smart handling of the VSAM datasets, the following procedures are provided:
ERBVSDEF

defines a Monitor III VSAM dataset

ERBV2S

unloads the VSAM data to a sequential dataset

ERBS2V

reconverts the sequential dataset to VSAM

Resource Measurement Facility

RMF Distributed Data Server


has Sysplex-wide scope
exploited by RMF PM, Monitor III Data Portal,

CIM Provider
talks XML over HTTP
no configuration needed!
just enter S GPMSERVE or F RMF,DDS on one

TCP/IP

image of the Sysplex

GPMSERVE
(RMF Distributed Data Server)

RMF Sysplex Data Server (ERB3XDRS Service)

RMF M III Data Gatherer


RMF M III Data Gatherer

11

RMF M III Data Gatherer

RMF Technical Overview

2010 IBM Corporation

The RMF Distributed Data Server (DDS) is the component, which serves as data source for the RMF client applications
There is just one additional address space (GPMSERVE) needed per Sysplex. The DDS uses the ERB3XDRS service to
retrieve Sysplex-wide Monitor III data.
In a mixed release configuration, the DDS should run on system with the highest RMF release

Resource Measurement Facility

Historical Reporting
Activities measured by Monitor I

Other Activities
Lotus Domino Server
HTTP Server

12

RMF Technical Overview

Lotus 1-2-3 and Excel

Spreadsheet Applications

XCF
OMVS
Coupling Facility
HFS

SMF

RMF Spreadsheet Reporter

Activities measured by Monitor III

RMF Data
Buffer

RMF Postprocessor

Cache Subsystem
Channel Path
CPU / Crypto
Device
Enqueue
Ficon Director
I/O Queuing
Page/Swap Data Set
Paging
Trace
Virtual Storage
Workload

2010 IBM Corporation

The tool of choice for historical reporting and after the facts analysis is the RMF Postprocessor
Accordingly to the Monitor I data collection options (there is one SMF record type for each resource), the reporting structure
is resource oriented (exception: the WLMGL record (SMF 72.3) holds performance data and resource consumption on
workload granularity)
Some SMF record types (74.x) are written by the Montior III data gatherer address space RMFGAT. However, this address
space should be active anyway to guarantee access to the Monitor III reports
With OS/390 R10 the Postprocessor has been enabled to accept foreign SMF records:
Type 103 HTTP Server
Type 108 Domino Server
While the RMF Postprocessor generates tabular reports, the RMF Spreadsheet Reporter can create graphical charts based
on Postprocessor data

Resource Measurement Facility

Postprocessor:
Postprocessor: Preparing SMF Data

//SMFDUMP EXEC PGM=IFASMFDP


//IDD1
DD
DISP=SHR,DSN=<input_smfdata_system1>
//IDD2
DD
DISP=SHR,DSN=<input_smfdata_system2>
//SMFDATA DD
DISP=(NEW,PASS),SPACE=(CYL,(10,10),RLSE),
//
UNIT=SYSDA,DCB=(RECFM=VBS,LRECL=32760,BLKSIZE=0)
//SYSIN DD *
INDD(IDD1,OPTIONS(DUMP))
INDD(IDD2,OPTIONS(DUMP))
OUTDD(SMFDATA,TYPE(70:78))

//RMFSORT EXEC PGM=SORT


//SORTIN
DD
DISP=SHR,DSN=<input_smfdata_system1>
//
DD
DISP=SHR,DSN=<input_smfdata_system2>
//SYSIN
DD
*
SORT FIELDS=(11,4,CH,A,7,4,CH,A),EQUALS
MODS E15=(ERBPPE15,36000,,N),E35=(ERBPPE35,3000,,N)

13









SMF data is kept in VSAM datasets


Postprocessor requires sequential format
use SMF dump utility IFASMFDP to unload the data
usuallly GDGs are the prefered target:
RMF.SMFDATA.SYSNAME(0)
RMF.SMFDATA.SYSNAME(-1)

SMF reords must be sorted by date and time


SORT step is required for sysplex-wide reporting
Workload Activity Report
Coupling Facility Report
Shared DASD Report

RMF Technical Overview

2010 IBM Corporation

Before the raw SMF data can be feeded to the Postprocessor, two preparation steps are needed:
The data has to be transformed from VSAM to Sequential format
This is achieved with the SMF Dump Utility IFASMFDP
In most installations, the SMF data is dumped by default to Generation Data Groups (GDGs)
For Sysplex-wide reporting, the records arriving from the various systems must be sorted by ascending interval time
Therefore, a preceeding SORT step must be executed
RMF provides the exit routines ERBPPE15 and ERBPPE35 to enable the sorting by RMF interval start time

Resource Measurement Facility

Postprocessor JCL

 DD Names

MFPMSGDS
MFPINPUT
PPRPTS
PPSUMnnn
PPXSRPTS
PPORPnnn
PPOVWREC

Message Output
SMF Input Datasets
combined Interval Reports
Summary Report Output
Sysplex Report Output
Overview Report Output
Overview Record Output

 Control Statements

14

DATE
RTOD
DINTV
REPORTS
OVERVIEW
OVW

Start / End Date


Start / End Time
Duration Interval Length
Report Types
Report or Record
Overview Control Statement

JCL can be generated


by ISPF Application or
Spreadsheet Reporter

//RMFPP
EXEC PGM=ERBRMFPP
//MFPMSGDS DD
DISP=SHR,DSN=*.ALLOC.MSG
//MFPINPUT DD
DISP=(OLD,DELETE),DSN=*.RMFSORT.SORTOUT
//PPRPTS
DD
DISP=SHR,DSN=RMF.INTERVAL.REPORTS
//PXSRPTS DD
DISP=SHR,DSN=RMF.SYSPLEX.REPORTS
//SYSIN
DD
*
DATE(04212003,04232003)
RTOD(0000,2400)
DINTV(0100)
REPORTS(ALL)

RMF Technical Overview

2010 IBM Corporation

Different DD names and control statements must be specified for the different kinds of report output
However, the user doesn't need to take care about any syntax: the JCL to execute the Postprocessor can be generated with
an ISPF dialog or with the RMF Spreadsheet Reporter workstation frontend

Resource Measurement Facility

Postprocessor:
Postprocessor: Standard Reporting
C P U

//RMFPP EXEC PGM=ERBRMFPP


//SYSIN
DD
*
DATE(10142002,10142002)
RTOD(1100,1300)
REPORTS(CPU)
SYSRPTS(WLMGL(SCPER))
SYSOUT(H)

z/OS V1R8

CPU

2084

SYSTEM ID SYSD
RPT VERSION V1R8 RMF
MODEL

314

H/W MODEL

---CPU--- ONLINE TIME


NUM TYPE PERCENTAGE
0
CP
100.00
1
CP
100.00
2
CP
100.00
3
CP
100.00
CP
TOTAL/AVERAGE

LPAR BUSY
TIME PERC
3.23
2.77
2.58
2.54
2.78

W O R K L O A D
z/OS V1R8

SYSPLEX SYSDPLEX
RPT VERSION V1R8 RMF

A C T I V I T Y

DATE 02/13/2007
TIME 11.00.00

INTERVAL 15.00.048
CYCLE 1.000 SECONDS

B16
MVS BUSY
TIME PERC
3.61
3.18
2.93
2.97
3.17

CPU SERIAL
NUMBER
066F7A
066F7A
066F7A
066F7A

I/O TOTAL
INTERRUPT RATE
14.60
22.53
25.61
29.42
92.16

% I/O INTERRUPTS
HANDLED VIA TPI
0.69
0.49
0.43
0.46
0.50

A C T I V I T Y

START 02/13/2007-11.00.00 INTERVAL 000.15.00


END
02/13/2007-11.15.00

MODE = GOAL

POLICY ACTIVATION DATE/TIME 02/12/2007 08.43.05


------------------------------------------------------------------------------------------------------------ SERVICE CLASS PERIODS
REPORT BY: POLICY=DEFAULT

-TRANSACTIONSAVG
23.36
MPL
23.36
ENDED
0
END/S
0.00
#SWAPS
106
EXCTD
0
AVG ENC
0.00
REM ENC
0.00
MS ENC
0.00

15

WORKLOAD=SYSTEM

TRANS-TIME HHH.MM.SS.TTT
ACTUAL
0
EXECUTION
0
QUEUED
0
R/S AFFIN
0
INELIGIBLE
0
CONVERSION
0
STD DEV
0

SERVICE CLASS=SYSTEM
CRITICAL
=NONE

--DASD I/O-SSCHRT 30.3


RESP
1.1
CONN
0.6
DISC
0.0
Q+PEND
0.3
IOSQ
0.1

---SERVICE---IOC
15835
CPU
5183K
MSO
0
SRB
1548K
TOT
6747K
/SEC
7496
ABSRPTN
TRX SERV

321
321

RESOURCE GROUP=*NONE

--SERVICE TIMES-CPU
26.498
SRB
7.913
RCT
0.017
IIT
0.275
HST
0.000
AAP
0.000
IIP
N/A
PROMOTED

PERIOD=1 IMPORTANCE=SYSTEM

---APPL
CP
AAPCP
IIPCP

%--3.86
0.00
0.00

-----STORAGE----AVG
2871.20
TOTAL
67066.09
SHARED
11.00

AAP
IIP

0.00
N/A

--PAGE-IN RATES-SINGLE
0.0
BLOCK
0.0
SHARED
0.0
HSP
0.0

0.000

RMF Technical Overview

2010 IBM Corporation

In most cases, the RMF Postprocessor is used to produce tabular reports (one per interval).
The chart shows an example for a single system report and for a Sysplex-wide report:
The CPU Activity Report shows the processor utilization from the systems perspective (on logical processor level)
The Workload Activity Report shows Sysplex-wide transaction and resource consumption statistics (on service class period
level)

Resource Measurement Facility

Postprocessor:
Postprocessor: Overview Reporting
OVERVIEW(RECORD,REPORT)
OVW(PROCS(NUMPROC))
OVW(CPUBSY(CPUBSY))
OVW(APPL(APPLPER(POLICY)))
OVW(TRANS(TRANS(POLICY)))
OVW(RTIME(RTIME(POLICY)))
DINTV(0100)

(via Spreadsheet Reporter)

R M F
z/OS

V1R8

NUMBER OF INTERVALS 4
DATE
MM/DD
08/07
08/07
08/07
08/07

TIME
HH.MM.SS
11.00.00
12.00.00
11.00.00
12.00.00

INT
HH.MM.SS
01.00.00
01.00.00
01.00.00
01.00.00

O V E R V I E W

SYSTEM ID SYSF
RPT VERSION V1R8 RMF

R E P O R T

START 08/07/2006-11.00.00
END
08/07/2006-15.00.00

INTERVAL 01.00.00
CYCLE 1.000 SECONDS

TOTAL LENGTH OF INTERVALS 04.00.00


PROCS

CPUBSY

APPL

TRANS

RTIME

5.0
5.0
5.0
5.0

9.2
8.5
8.3
8.8

33.9
33.0
30.6
24.5

16.8
28.1
38.9
24.5

40.56
35.92
25.66
44.99

Format of the INTERVAL column


changes from MM.SS to HH.MM.SS

16

RMF Technical Overview

2010 IBM Corporation

When the user wants to focus on specific metrics without analyzing an entire report, the Postprocessor Overview feature
provides a smart solution: by means of overview control statements, he can extract specific values and produce his own
customized reports
Multiple intervals are visible at a glance: Thus, Overview reporting is well suited for trend analysis over long periods
For all SMF 70-78 record types, the user can choose from a rich set of Overview Control statements. For more details, refer
to the RMF Users Guide (SC33-7990), Chapter 17, Overview and Exception Conditions

Resource Measurement Facility

RMF Spreadsheet Reporter

17

converts SMF Data to Spreadsheet Format


creates graphical Views for Trend Analysis
can be downloaded from the RMF Homepage

RMF Technical Overview

2010 IBM Corporation

In extension to the RMF Postprocessor, RMF offers a graphical workstation frontend for historical reporting:
the RMF Spreadsheet Reporter
The visualization of the data is performed by spreadsheet macros, which are shipped as samples together with the
Spreadsheet Reporter main application.
The Spreadsheet Reporter is a basic part of the RMF product and doesnt require additional charges. The installable image
is shipped with the host file SYS1.SERBPWS(ERB9R2SW) or the most recent version can be downloaded from the RMF
homepage: http://www.ibm.com/servers/eserver/zseries/zos/rmf/. You will find the Spreadsheet Reporter in the Tools
category of the homepage.

Resource Measurement Facility

RMF Spreadsheet Reporter

SMF raw Data


RMF Postprocessor

Report Listings

Overview Records
Download

Report Listings

Overview Records
Extract

*.RPT Files

*.OVW Files
Convert

CPU Contention: System SYS1

*.WK1 Files
Generate

Working Sets
Display

Spreadsheet
18

RMF Technical Overview

2010 IBM Corporation

The transition from the SMF raw data to graphical charts is a complex process with multiple steps.
The good news - the RMF Spreadsheet Reporter does it all for you by magic:

Generates Report Listings or Overview Records from the SMF data with the RMF Postprocessor: on the Resource
notebook page of the Spreadsheet Reporter, specify as input the SMF dump data sets from all the systems of your
sysplex. Then check the desired reports on the Reports selection dialog. Now you are ready to exploit the Spreadsheet
Reporters remote job execution engine, which allows you to run the Postprocessor from your workstation!

Downloads the Postprocessor output to your workstation

Extracts the reports or records

Convert the reports or records to spreadsheet format and generates the Working Set

Resource Measurement Facility

RMF Spreadsheet Reporter

Macros
for all Types of
Performance relevant
Areas !!

19

RMF Technical Overview

2010 IBM Corporation

Once a Working Set has been created from SMF data, the user can select from a rich set of macros for all performance
relevant areas
Three basic types of macros can be distinguished:
Macros based on data contained in one Postprocessor Report (for detailed analysis of one single interval)
Macros based on data contained in Overview Records (for trend analysis with multiple intervals)
Macros based on data contained in multiple Postprocessor Reports (hybrid macros)
The new XCF Trend Report macro belongs to the family of hybrid macros

Resource Measurement Facility

Realtime Reporting

CF




covers all Sysplex related aspects


two monitors and a workstation extension
Monitor

III, best suited for

short-term, real-time and historical reporting


online performance analysis
goal attainment supervision
sysplex-wide and single-system reporting
monitoring of exceptional conditions

WLM

Monitor

II, best suited for

snapshot reporting
single job and resource monitoring

RMF

PM / Data Portal

enterprise-wide reporting of z/OS systems


based on RMF Monitor III data

20

RMF Technical Overview

RMF offers three different Monitors for realtime reporting:


Monitor III Reporter (ISPF):
Is designed for workflow analysis and goal attainment supervision
Can be used in realtime mode, but supports historical reporting as well
Monitor II (ISPF and native TSO):
Is a snapshot monitor. The interval length can be determined by the user
Consists of a set of address space and resource oriented reports
RMF Performance Monitoring / Monitor III Data Portal:
Allows to manage multiple Sysplexes from the workstation
The raw data is identical with Monitor III ISPF, but the GUI is totally different

2010 IBM Corporation

Resource Measurement Facility

Monitor III Reporting


Monitor III Delay Monitoring






Processor
Storage
Device
Enqueue
Operator
Message
Tape Mount
Subsystem
HSM - JES - XCF

Monitor III Features


Monitor III Activity Monitoring















21

Common Storage
Page/Swap Data Sets
Storage Frames
Device
Data Set Level by Job and Volume
Cache
Coupling Facility
Goal Attainment
VSAM RLS
UNIX System Services
Enclaves
zFS
Diskspace
Spin/Suspend Locks














Cursor-Sensitive Navigation
Workflow/Exceptions Monitoring
Automatic Customization
Support of WTO Messages
Continuous Monitoring
Hardcopy Reports
On-Line Tutorial
On-Line Help
Adaptive Reports
User Reports
Sysplex-wide Reports
Remote Reporting

RMF Technical Overview

2010 IBM Corporation

The Monitor III ISPF Reporter provides two basic types of monitoring:
WorkloadDelay monitoring
The state samplers are inspecting all hard- and software resources and assign those resources on address space
level
Possible states for an address space are:
Delayed for Processor, Storage, Device, Enqueue, Operator, Subsystem
Using Processor or Device
Idle (timer wait or terminal input wait)
Unknown (e.g. using an unmonitored waiting mechanism)
ActivityResource Monitoring
In addition to the workloaddelay concept, there are lots of resource oriented reports with similar statistics than the
RMF Postprocessor reports
Together with hardware or software related innovations new reports have been added with each z/OS release

Resource Measurement Facility

States of a Job
Proc

Idle

Dev

Dev
Dsn

Dsn
JES

Enq
Oper

HSM

Stor

XCF

Proc

?
Unknown

22

Using(%) =

using samples
number of samples

100

= 50%

Delay(%) =

delay samples
number of samples

100

= 33%

Workflow (%) =

using samples
using samples + delay samples

RMF Technical Overview

100

2010 IBM Corporation

Address spaces are either using resources or they are delayed for resources or they are idle
The RMF state samplers are trying to monitor continuously all kinds of hard- and software resources
Depending on the result, all address spaces can be associated with the following qualities:
Using%
Delay%
Workflow%
Since not all address spaces can be catched up every cycle, the remaining states are counted as unknown. Possible
reasons for unknown are:
The address space is delayed for unmonitored resource (other than DASD or Tape)
The address space is using an unmonitored waiting mechanism (most Started Tasks!)
statistical reasons

= 60%

Resource Measurement Facility

Example: Using and Delay

D
B

Job
A
B
C
D

Res

D
*
*

Why
C
C

*
*

Job
A
B
C
D

Res

D
*

Why
B

*
*
*

Job
A
B
C
D

U
*

Why

D
*

Why
C

*
*

RMF Monitor III Resource Delay Report


Samples: 4

Jobname

WFL USG DLY IDL Primary


%
%
%
% Reason

WFL
Resource %

C
C
A
C

Job
A
B
C
D

Time: 06.28.20 Range: 4 Sec

25 25 75 0
50 25 25 50
66 50 25 25
0 0 25 75

Res

Samples: 4

A
B
C
D

Res

RMF Monitor III Delay Report

23

Res

Time: 06.28.20 Range: 4 Sec


ADU

40 1.5

Jobname USG DLY Reason


%
%

A
B
C
D

25
25
50
0

75
25
25
25

C
C
A
C

2010 IBM Corporation

RMF Technical Overview

The example helps to understand what is calculated and reported by RMF Monitor III
From the jobs perspective:
Job A

1x Using(25%)

3x Delayed(75%) 0x Idle(0%)

Workflow: 1/4 = 25%

Job B

1x Using(25%)

1x Delayed(25%) 2x Idle(50%)

Workflow: 1/2 = 50%

Job C

2x Using(50%)

1x Delayed(25%) 1x Idle(25%)

Workflow: 2/3 = 66%

Job D

0x Using(0%)

1x Delayed(25%) 3x Idle(75%)

Workflow: 0/1 = 0%

Workflow = 4/10 = 40 %

Average Delayed User (ADU): 6/4 = 1.5

From the resource perspective:


4x using, 6x delay

Resource Measurement Facility

Monitor III: Job Delays


RMF V1R7
Samples: 100

System: AQTS

Delay Report
Date: 07/18/06

Line 1 of 326
Time: 06.28.20

Range: 100

Sec

24

Name

Service
CX Class

SUSANK
CONSOLE
RRSSERVQ
BHBE
MORABIT
RONDA2A
D24JAP1
RRSSERVQ
GRSARTSQ
RRSSERVQ
CATALOG
ANFWPROC
GRSARTSQ
SMFDRS
JES2
GRSARTSQ
ARTXESQ
DFRMM

T
S
B
T
T
B
T
B
B
B
S
SO
B
S
S
B
B
S

Cr

WFL USG DLY IDL UKN ---- % Delayed for ---- Primary
%
%
%
%
% PRC DEV STR SUB OPR ENQ Reason

TSOPRIME
SYSTEM
WLMSHORT
TSOPRIME
TSOPRIME
COMBUILD
TSOPRIME
WLMSHORT
WLMSHORT
WLMSHORT
SYSTEM
SYSSTC
WLMSHORT
STCMED
SYSSTC
WLMSHORT
WLMSHORT
SYSSTC

0
0
0
40
41
42
49
50
50
50
63
67
71
71
73
80
80
83

0 100
0 15
0
1
4
6
37 56
29 42
22 23
2
2
1
1
1
1
57 39
2
1
5
2
5
2
16
6
4
1
4
1
81 18

0
0
0
90
0
0
56
0
0
0
0
0
0
0
0
0
0
0

0
85
1
0
7
29
0
0
0
0
12
2
0
93
79
0
0
1

0
0
0
6
0
1
0
0
0
0
0
0
0
0
0
0
0
0

0
0
0
0
56
41
23
2
0
0
39
1
1
0
6
0
1
18

0 100
0
0
0
1
0
0
0
0
0
0
0
0
0
0
0
1
0
1
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0

0
15
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0

0
0
0
0
0
0
0
0
0
0
0
0
0
2
0
0
0
0

Address Space
Performance at a Glance !
sorted by ascending Workflow
Delay Type Breakdown
Delay Reason Information

HSM
Message
JES
JHUGO
COMPK5
COMPK5
PRIPK5
SPOL1J
JES
JES
MCATTS
SPOL1J
SPOL1J
SYSZVVDS
SPOL1J
JES
SPOL1L
SL3061

RMF Technical Overview

2010 IBM Corporation

The Address Space Delay report is the basic report that visualizes the overall Monitor III concept
The report is sorted by ascending workflow: thus, the critical candidates with low workflow appear on top of the address
space list
Beyond the basic statistics (workflow, using, delay, idle, unknown), a delay type breakdown for the following resources is
provided:
PRC

processor

DEV

device

STR

storage

SUB

software subsystem

OPR

operator

ENQ

enqueue

Additionally, the delay reason information is displayed in the rightmost column:


For DEV the critical volume
For PRC the competing address space with the highest usage
For SUB the subsystem name
For ENQ the enqueue resource name

Resource Measurement Facility

Monitor III: Workflow/Exceptions


RMF V1R7
Samples: 100

System: AQTS

Workflow/Exceptions
Date: 07/18/02

Time: 06.28.20

Line 1 of 32
Range: 100

Sec

--------------------------- Speed (Workflow) --------------------------------Average CPU Util: 81 %


Speed of 100 = Maximum, 0 = Stopped
Name
Users Active
Speed
Name
Users Active
Speed
*SYSTEM
560
35
42
*DEV
53
15
46
ALL TSO
19
4
17
*MASTER*
1
1
100
ALL STC
485
6
52
ALL BATCH
39
24
37
ALL ASCH
Not avail
ALL OMVS
17
1
92
*PROC
292
8
43
------------------------------ Exceptions ------------------------------------Name
Reason
Critical val. Possible cause or action
*SLIP*
SLIP PER TRAP
SLIP ID=U427 is active.
ALL TSO
DEV -RMFLIB
1.5 users
Short block size or excessive seeks.
*SYSTEM
PROC-SUSANK
12.1 users
BHBE
SUBS-JES
100.0 % delay Processing TSO OUTPUT command request.
MORABIT
PROC-SUSANK
75.0 % delay
GANDALF
DEV -SMS001
20.0 % delay Device may be overloaded.
CONSOLE
OPER-Message
13.0 % delay Awaiting reply to operator request 52.
HSM
Not avail
Job HSM is not running.
SYSPAG
Not avail
Volume SYSPAG is not mounted.

25

Bottleneck Detection
at a Glance !
Workflow for Groups and
Resources
Exception Lines for individual
Conditions

RMF Technical Overview

2010 IBM Corporation

The Monitor III Workflow Exception Report consists of two independent sections:
The Workflow section provides an overview about groups of work, individual address spaces or resources
The Exceptions section displays alerts when critical thresholds are exceeded
The content of both sections can be customized individually:
For the Workflow section, up to 14 candidates for continuous display of workflow can be selected
For the Exceptions section, alerts for almost all Monitor III metrics (as well as combination of metrics) can be defined

Resource Measurement Facility

Monitor III: Goal Attainment


RMF 2.10.0 Sysplex Summary - SCLMPLEX
Command ===>
WLM Samples: 240

Systems: 3

Date: 05/15/02 Time: 13.00.00 Range: 60

Service Definition: SCLM


Active Policy: STANDARD

Name

STC
STCCMD
SYSTEM
SYSSTC
SYSTEM
TSO
PRDTSO

W
S
W
S
S
W
S
1
2
3
R

MASTER

26

1
1
1

40

0.46
N/A
N/A

1.000 AVG
1.500 AVG
2.000 AVG
N/A

Sec

Installed at: 12/06/00, 10.07.24


Activated at: 12/06/00, 10.07.33

------- Goals versus Actuals -------Exec Vel --- Response Time --- Perf
Goal Act ---Goal--- --Actual-- Indx
88
88
69
N/A 68
N/A 70
84
84
60
0.0
85
N/A 47

Line 1 of 14
Scroll ===> CSR

0.080
0.109
2.928

AVG
AVG
AVG

0.08
0.07
1.46

Trans --Avg. Resp. TimeEnded WAIT EXECUT ACTUAL


Rate
Time
Time
Time
0.000
0.000
0.000
0.000
0.000
2.100
2.100
1.150
0.567
0.383
0.000

0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000

0.000
0.000
0.000
0.000
0.000
0.608
0.608
0.080
0.109
2.928
0.000

0.000
0.000Sysplex Performance at a Glance
0.000 80 Intervals in GO Mode
0.000
0.000 colored Indication for PI > 1
0.608Importance = 1+2
0.608
0.080Importance > 2
0.109
2.928
0.000

RMF Technical Overview

2010 IBM Corporation

The Sysplex Summary Report is the standard report for goal attainment supervision
The report provides Goals vs. Actuals on Service Class Period level. In case the Performance Index (PI) for a certain
Service Class Period is higher than 1, coloured lines are generated:
A red line for Service Class Periods with high Importance (1, 2)
A yellow line for Service Class Periods with low Importance (3, 4, 5)
In GO Mode, a coloured bar shows the state for the last 80 intervals: when at least one line for the current interval becomes
red or yellow, a red or yellow square will be added. Otherwise the bar is extended with a green square. Thus, the Sysplex
performance is visible at a glance for an 80-minute period (proposed a 60 second reporting range)
For Service Classes with response time goals, the report provides a direct link to Response Time Distribution Report by
means of cursor sensitivity

Resource Measurement Facility

Monitor III: Session Setup


VSAM Archives
alloc f(rmfds00) da(rmf.m3.sys1.data)
alloc f(rmfds01) da(rmf.m3.sys2.data)

TSO Address Space


VSAM Datasets
SYS1.SERBCLS
(ERBRMF3X)

Monitor III
Reporter
Type text

Type text

ERBTLIB
&USER.&REL.ISPTABLE
SYS1.SERBTENU
SYS1.SERBT

Type text

RMFGAT
In Storage Buffer

&USER.&REL.ADMGDF

27

RMF Technical Overview

2010 IBM Corporation

The options for a Monitor III reporter session are stored in tables (e.g exception definitions). The initialization procedure
ERBRMF3X performs the table allocations when a reporter session is going to be started
By default, a reporter session accesses directly the actual data collected by the RMFGAT (via Cross Memory)
However, it is also possible to establish a standalone reporter session independent from the Monitor III gatherer: in order to
analyze historical data, up to 99 VSAM datasets can be preallocated (DDNAME = RMFDSxx)

Resource Measurement Facility

Monitor III Reporter Usage


RMF

Session
Color
etc.

RMF
Main Menu

Report
Options

Session
Options

RO
F3

M III
Main Menu

Tutorial

Help

Report
Selections,
Short
Commands

F1

System Information
Samples: 100 System: AQTS
Group
WFL
*SYSTEM 30
*TSO
20
DMN001
28
PG002
20

User
250
22
16
22

...
...
...
...
...

Enter
F3

Scroll

Monitor II

Enter

Tutorial

F7

Postprocessor
Selection Menu

F8

BREF/FREF
F3

F10

F11

Find, Color,
ICU, Current,
GO,
and more ...
28

RMF Technical Overview

The Monitor III reporter session can be invoked directly from the RMF Main Menu
Various session options (SO command) can be defined and stored persistently, for example:
initial mode (GO or STOP)
initial screen
range length
By means of report options (RO command), the content of almost all reports can be customized
A very detailed tutorial provides lots of information about the Monitor III concepts and features
Cursor sensitive HELP (including detailed metric descriptions) is always available via the PF1 key

2010 IBM Corporation

Resource Measurement Facility

Monitor III Report Overview

Sysplex Reports

Coupling Facility

SYSSUM

CFSYS

SYSRTD

CFOVER

SYSWKM

CFACT

Cache Subsystem

VSAM RLS

RLSSC

CACHSUM

User Reports

SYSENQ

RLSDS
CACHDET

DSINDEX

RLSLRU

RG

System/Workload Reports

Monitoring Reports

GROUP

SYSINFO

CPC

WFEX

Delay Reports

Job/Workload oriented

PROC

DEV

DSND

DSNV

STORS

HSM

ENCLAVE

MSI

OPD

SYSTREND

DELAY

Resource Utilization Reports

DEVN

STORCR

DEVT

STORC

DSD

Resource oriented

ENQR

ENQ
STOR

Other

DEVR
STORR

STORF
STORCR

JES
CHANNEL

PROC

XCF
Job
Reports

IOQUEUE
Modified panel
User-written report
Sortable report

Detailed Job Analysis

29

JOB

RMF Technical Overview

2010 IBM Corporation

Monitor III offers a complete report suite which covers all performance relevant areas. The reports can be grouped by
categories:
A family of reports with sysplex wide scope
System level reports for various kinds of measurements
Job/address space oriented reports
Reports from the perspective of the major hard-and software resources
User reports which can be adapted or customized
In order to improve the Monitor III capabilities, new reports have been added frequently, e.g.
OS/390 R4

DSND DSNV

OS/390 R6

CFSYS CFOVER CFACT

OS/390 R7

CACHESUM CACHDET ENCLAVE

OS/390 R10

RLSSC RLSDS RLSLRU OPD

z/OS V1R2

CPC

z/OS V1R7

ZFSSUM ZFSACT SPACEG SPACED

z/OS V1R8

XCFOVW XCFSYS XCFPATH XCFGROUP (Data Portal only)

Resource Measurement Facility

Monitor III: Setup for WTO


WTOs
 Monitor III Batch Address Space creates Reporting Tables

Monitor III
Gatherer

 Thresholds can be defined via


Workflow Exception Options Dialog
Reporter Phase Exit Module
 Console Message is generated by Exit Module (e.g. WLM Capping)

$HASP100 BMAI
ON TSOINRDR
$HASP373 BMAI
STARTED
IEF125I BMAI - LOGGED ON - TIME=12.36.20
+RMF300I 3B: Processing CPC Report...
+RMF301I 3B: Local Partition Capping State:
+RMF303I 3B: Time until Capping (sec):
40 (WTO Limit:
+RMF304I 3B: MSU Consumption of critical LPARs:
+RMF305I 3B: SYS1 :
64 (WTO Limit: 60)
+RMF305I 3B: SYS4 :
48 (WTO Limit: 32)
IEF126I BMAI - LOGGED OFF - TIME=12.38.00
$HASP395 BMAI
ENDED
$HASP250 BMAI PURGED
+RMF300I 3B: Processing CPC Report...
+RMF301I 3B: Local Partition Capping State:
+RMF302I 3B: WLM Capping %: 24.2 (WTO Limit: 10.0)
+RMF304I 3B: MSU Consumption of critical LPARs:
+RMF305I 3B: SYS1 :
82 (WTO Limit: 60)
+RMF305I 3B: SYS2 :
12 (WTO Limit: 10)
$HASP100 BMGU
ON TSOINRDR
$HASP373 BMGU
STARTED
IEF125I BMGU - LOGGED ON - TIME=12.39.40

30

Set of
Samples

Monitor III
Batch
Reporter
600)

RMF Technical Overview

WTO
Exit
Module

Sample Exits provided for:


Workflow Exception Report
Sysinfo Report
CPC Capacity Report

2010 IBM Corporation

The procedure RMFM3B is supplied to run a Monitor III reporter session in batch mode. This environment is well suited,
when a threshold is exceeded and a corresponding message should be displayed on the console
For threshold processing, the user can select from all metrics contained in one of the Monitor III reports: the procedure
accepts any report name as input parameter. In other words, the specified report is the one and only report that is produced
by the batch reporter session (multiple reports are not supported)
The scanning of the report tables and the comparison against thresholds is performed by exit routines. Depending on the
result of this comparison, a WTO can be generated
RMF ships the following REXX exit routines as examples (in SYS1.SERBCLS):
ERBR3WFX

processes the WFEX report

ERBR3SYS

processes the SYSINFO report

ERBR3CPC

processes the CPC report

Resource Measurement Facility

Monitor II: Overview


 Monitor II is a Snapshot Reporter
collects the status of system resources (CPU, devices, paging activity, ...)
collects the status of address spaces (resource usage, state information)

 use Monitor II to

continuously monitor resource usage


determine the state of any address space in the system
track CPU usage of problem address spaces
collect supplemental information when analyzing performance problems with Monitor III

 choose Background Session


to collect SMF records for archiving and later postprocessing
to automate snapshot reporting

 choose Display Session

Postprocessor

for immediate feedback


for online analysis
f rmf,s aa

10:12:08
JOBNAME
*MASTER*
PCAUTH

31

DEV
CONN
2618
1264

SMF 79.x

SMF 79.x

RMF Technical Overview

2010 IBM Corporation

Monitor II complements the host based RMF reporting suite. It is the RMF snapshot reporting feature
Three basic groups of reports can be distinguished:
Address space oriented reports (e.g. ARD, ASD, ASRM)
I/O subsystem related reports (e.g. DEV, CHANNEL)
Resource reports (e.g. SENQ, SPAG)
Monitor II is the appropriate function, when immediate feedback about address spaces or resources is required
However, by means of a background session, Monitor II can also produce SMF Type 79 records. These records can be
feeded to the RMF Postprocessor for after the facts analysis

Resource Measurement Facility

Monitor II Reporting

Activities measured by Monitor II:


 Address Space Data













32

Resource
State
SRM
Channel Path
Device
I/O Queuing
Enqueue
HFS
IRLM Long Locks
Paging
Page/Swap Data Set
SRM Resource Data
Sysplex Data Server
Library Display

ISPF

for most comfortable usage


supports sorting and finding
started from TSO READY or

from RMF main menu

TSO

don't use it anymore!

3270

dropped with z/OS V1R2

RMF Technical Overview

2010 IBM Corporation

Monitor II collects statistics about activities and resources (but no delays like Monitor III)
The most heavily used Monitor II reports are the address space oriented reports
Address Space Resource Data(ARD): Resource consumption of address spaces (CPU, I/O Storage)
Address Space State Data(ASD): State Information for address spaces (Service Class, Dispatching Priority, Location
e.g IN or OUT)
Address Space SRM Data(ASRM): Details from the SRM point of view (Transaction Active Time, Service Units...)
ISPF is the recommended Monitor II reporting environment (although Monitor II reports can be invoked from native TSO as
well)

Resource Measurement Facility

Monitor II Commands
TSO-Command: RMF MON2

e.g. ASD report options


Find string
RFind

Cancel
Report Options
e.g. ASD

Monitor II Primary Menu

Enter selection number or command

RMF - ASD

e.g. ARD

RMF
Selection ===>

Reset

Selection

RO

RMF - ARD
Command ===>

report-name

F7

Scroll up

F8

Scroll down

MIG=63.1K
1 Address Spaces
2 I/O Subsystem
3 Resource
L Library Lists
U User

Return

F4

11:15:27 DEV
JOBNAME CONN
*MASTER*
PCAUTH
TRACE
DUMPSRV

364.3
0.000
0.000
1.113

Other commands:
Sort
H
D
Print
Sys
Keys

33

Prints all reports


Delta-mode
Prints current screen
Remote reporting
View/Assign PF-keys

F6

FF PRIV LSQA
BEL
FF CSF
0
0
0
0

455
2
3
2

106
32
105
62

GO nn for auto update

<Enter> for manual update

RMF Technical Overview

2010 IBM Corporation

Monitor II reports can be selected either by menu choice or by report name


With the report options command RO, the report contents can be customized (e.g. Workloads TSO, Batch....)
Use the PF7/PF8 keys for scrolling (in case not all report lines are visible)
Note, that all reports are sortable (place the cursor on the desired coulmn header and press PF6)
The GO command provides the automatic refresh of the report data
With the SYS command, the reports from all remote systems in the Sysplex can be accessed

Resource Measurement Facility

Monitor II: ARD Report


RMF - ARD Address Space Resource Data

Line 1 of 48
Scroll ===> PAGE

Command ===>
CPU= 45/ 30 UIC=2540 PR=
16:04:11 DEV
JOBNAME CONN
HSM
ANTMAIN
*MASTER*
XCFAS
IOSAS
RMFGAT
JES2
SMSVSAM
SMS
OMVS
CATALOG
HSMMON
NET
SMF
LLA
NETVSCLM
IXGLOGR

79920
28880
8616
4485
3774
3144
773.4
771.2
697.1
548.4
376.5
81.79
66.93
52.73
27.73
24.02
23.65

FF PRV LSQA LSQA X


SRM
16M FF CSF ESF M CR ABS
1
4 136
2 23 160
0 731 102
0 2K 1059
0 29 109
1 20
65
9 66 185
1 205 764
0
2
53
5 131 630
0
2 234
0
3
53
0 36
79
0
2
56
0 40
67
1
5 148
0
5
87

6
12
18
459
10
6
43
85
12
60
1
9
67
8
14
31
48

X
X
X
X
X
X
X
X
X
X
X
X
X

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

TCB
TIME

8508
360.2
296.6
409.7
46.07
5160
360.8
434.8
166.1
85.77
111.7
99.15
771.0
0.20
3.65
115.1
38.28

System= SCLM Total

CPU EXCP SWAP LPA CSA NVI V&H


TIME RATE RATE RT RT RT RT

10575
427.7
2652
796.5
46.89
5236
455.3
550.2
174.2
103.0
117.4
101.8
1556
3.70
4.09
125.9
45.04

0.01
0.00
0.04
2.41
0.02
0.67
2.19
1.55
2.88
0.00
0.21
0.19
0.00
0.00
0.06
0.00
0.00

0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0

Address Space Resource


Consumption at a Glance !
I/O Activity
Frame Counts
CPU Time

all Table Reports


are sortable !

34

RMF Technical Overview

2010 IBM Corporation

The ARD report provides a quick check about address spaces and their resource usage (e.g. Device Connect Time or TCB
Time)
The report header displays the following information:
MVS view of the CPU Utilization
LPAR view of the CPU utilization
Unreferenced Interval Count
Pagein Rate
The tabular part of the report contains similar information than the SDSF DA screen

Resource Measurement Facility

Monitor II Session Setup


H | Print

RMF - ARD
Command ===>

1. ALLOC F(RMFDMTSO)

DA('BHBE.M2.REPORTS') disp
DCB=(RECFM=VBA,BLKSIZE=1693,LRECL=137)

2. RMF MON2

MIG=63.1K
11:15:27 DEV
JOBNAME CONN
*MASTER*
PCAUTH
TRACE
DUMPSRV

364.3
0.000
0.000
1.113

FF PRIV LSQA
BEL
FF CSF
0
0
0
0

455
2
3
2

106
32
105
62

ERBROPT2
Report options

BHBE.RMFOS260.ISPTABLE

BHBE.M2.REPORTS

35

RMF Technical Overview

In case the user wants to save the reports for later analysis he can use the Hardcopy and Print commands
The reports are written either to SYSOUT or to the preallocated dataset RMFDMTSO
For most of the reports you can specify options or filters, e.g.
ARD:

address space types BATCH or TSO only

DEV:

volume names, device numbers or storage groups

For persistence, the report options are written to the dataset &uid.RMF&rel.ISPTABLE

2010 IBM Corporation

Resource Measurement Facility

Monitor II Display Modes


Specific resource or job, e.g. job
BHOL

All resources or jobs at a specific


point in time

RMF - ARDJ Address Space Resou


Command ===>

RMF - ARD Address Space Resour


Command ===>

CPU= 19/ 19 UIC=2540


BHOL
TIME

DEV
CONN

13:16:46
13:16:47
13:16:48
13:16:49
13:16:50
13:16:51
13:16:52
13:16:53
13:16:54

16.87
16.88
16.89
16.90
16.91
16.92
16.93
16.94
16.95

FF PRIV LSQA LSQA X SRM


BEL
FF CSF ESF M ABS
3
3
3
3
3
3
3
3
3

3
3
3
3
3
3
3
3
3

79
79
79
79
79
79
79
79
79

0
0
0
0
0
0
0
0
0

65K
65K
65K
65K
65K
65K
65K
65K
65K

TCB
TIME
7.20
7.20
7.20
7.21
7.21
7.21
7.22
7.22
7.22

CPU=
13:20:26 DEV
JOBNAME CONN
*MASTER*
PCAUTH
RASP
TRACE
DUMPSRV
XCFAS
GRS
SMXC
SYSBMAS

36

Current status in highlighted line

8 UIC=2540

FF PRIV LSQA LSQA X SRM


BEL
FF CSF ESF M ABS

409.1
0.000
0.000
0.000
1.113
273.9
0.000
0.000
0.000

Row Report

9/

0 503
0
2
0
4
0
3
0
2
0 1420
0
40
0
2
0
41

106
32
12
105
62
749
692
33
73

0
0
0
0
2
5
0
0
0

TCB
TIME

0.0 34.00 2
X 0.0
0.00
0.0
0.00
X 0.0
0.00
0.0
0.05
X 0.0 28.64
X 0.0 241.01 3
0.0 30.28
0.0
3.94

Table Report

Current status in whole report

RMF Technical Overview

The address space and volume oriented reports are available in two flavours:
Table reports:

one sample with all address spaces or all volumes at a glance is displayed

Row reports:

multiple samples for one address space or one volume are displayed

2010 IBM Corporation

Resource Measurement Facility

Monitor II Report Overview


Name

Mon I

Row

ISPF

ARD

Address space resource data

ARDJ

ASD

Address space state data

ASDJ

ASRM

Address space SRM data

ASRMJ

CHANNEL

Channel path activity data

DEV

DEVV

Device activity data


Y

HFS

ILOCK
IOQUEUE

I/O queuing activity data


Library lists

Page/swap data set activity

SDS

Sysplex data server statistics

SENQ

Enqueue contention

SENQR

37

HFS statistics
IRLM locking data

LLI
PGSP

Explanation

Enqueue reserve activity

SPAG

Paging activity

SRCS

Central storage, processor, SRM

RMF Technical Overview

2010 IBM Corporation

A subset of Monitor II reports is based on Monitor I gatherer data (DEV, IOQUEUE, PGSP).
Those reports are not available when the ZZ session is inactive (reason: Supervisor state and Key zero is required for
certain gathering activities)
The SORT capability as well as the HFS report is only available in the ISPF environment

Resource Measurement Facility

RMF Performance Monitoring

Enterprise-wide performance monitoring

38

of z/OS hosts
Platform independent Java Edition
Linux gathering support
Graphical user interface
Flexible definition of data
Persistent definition of views
Powerful data reduction
Analysis support

RMF Technical Overview

2010 IBM Corporation

RMF Performance Monitoring (RMF PM) is the graphical workstation frontend for Monitor III data. It allows to manage
multiple Sysplexes from one single point of control
RMF PM is free of charge. The most recent version can be downloaded from the RMF homepage:
http://www.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/pmweb/pmweb.html

Resource Measurement Facility

RMF PM: Perfdesk Concept

Perfdesk Folder
Perfdesk 1
Dataview 1

Dataview 2

Series 1
Series n

Series 1
Series n

Dataview 3

Dataview 4

Series 1
Series n

Series 1
Series n

Perfdesk 2

39

RMF Technical Overview

Dataview 1

Dataview 2

Series 1
Series n

Series 1
Series n

2010 IBM Corporation

The main window is of the RMF PM application is divided into 2 parts:


The left side contains the PerfDesk notebook page, which allows to control the content of the right side of the window
The right side contains the Open Perfdesk window: one Perfdesk consists of one or more views with the actual data
(Data Views). The data itself is represented by the so called Series
On the host side, the Distributed Data Server (DDS) address space must be started in order to provide the data for the PM
client by means of a TCP/IP connection

Resource Measurement Facility

RMF PM: Resources and Metrics

Sysplex

MVS Image
I/O Subsystem
All SSIDs
SSID
All LCUs
LCU
All Channels
Channel
All Volumes
Volume
Processor
Storage
Auxiliary Storage
Central Storage
CSA, SQA,
ECSA, ESQA
Expanded Storage
Enqueue
Operator
Subsystems
JES, XCF, HSM
CPC
LPAR
Coupling Facility
CF Structure

40

RMF Technical Overview

2010 IBM Corporation

Resources are components of a computing system. The Monitor III data model maps those resouces in terms of a
hierarchical scheme
Each resource is associated with a set of metrics. Individual metrics (or counters) can be selected for data collection and
continous display

Resource Measurement Facility

RMF PM: Intelligent Analysis


data views with click-sensitve bars
link to predefined Analysis PerfDesks

41

RMF Technical Overview

2010 IBM Corporation

One of the most powerful features of RMF PM is the Intelligent Analysis: once the user has detected an abnormal value, he
can navigate to context information or additional views in order to detect the reason of a problem. This is achieved by clicksensitive bars whereby the analysis menu choice offers a selection of various pathes for further drill-down

Resource Measurement Facility

Linux Performance Monitoring

RMF PM Java TM
Technology Edition - Linux
support available on Internet

Common View
for
z/OS and Linux
Data

42

RMF Technical Overview

2010 IBM Corporation

RMF PM has been extended to support the Linux operating system using the existing RMF PM infrastructure. Thus, the PM
capabilities (spreadsheet export, filter, historical data collection, graphical trend reports, etc.) are also available for Linux
environments
For the RMF PM client, it is transparent whether the host is a z/OS or Linux system
On the server side, two additional components are needed:
The Linux compiled version of the Distributed Data Server infrastructure
The Linux data gatherer backend (collects data out of the procfile system)
The Linux support can be downloaded free of charge: http://www.ibm.com/servers/eserver/zseries/rmf/pmlin.htm
Please note that the Linux support is not an official part of z/OS RMF

Resource Measurement Facility

RMF Monitor III Data Portal


direct connection to the RMF Distributed Data Server
just specify http://<hostname>:8803
all resources and metrics supported (same than RMF PM)

43

RMF Technical Overview

2010 IBM Corporation

Since the RMF Distributed Data Server talks XML over HTTP, the client can be either a fat client like RMF PM or the data
can also be accessed directly by means of a web browser. Thats what we call the RMF Monitor III Data Portal
In other words, the RMF DDS server behaves like a webserver: the data can be requested by means of the standard HTTP
protocol. In this case, the presentation is managed by associated formatting instructions in terms of Style Sheets (XSL
documents)
The chart shows the Overview Performance Desktop (=PerfDesk) with the following metrics:
% processor utilization by MVS image
% total physical utilization (CP) by partition
performance index by important WLM service class period
% CSA utilization by MVS image

Resource Measurement Facility

RMF Monitor III Data Portal


Portal

select favorite
Metric from List

click
Add this Metric to My View

for Persistence

select any Resource


from Configuration View

Coupling Facility PerfDesk with:


Request rates
Service times

44

RMF Technical Overview

2010 IBM Corporation

Beyond the Overview PerfDesk, the user can customize his own private PerfDesk (aka MyView). Thereby he can select
from a rich set of metrics for any kind of resource he wants to monitor contionuously. The definitions are stored persistently
in terms of browser cookies
The example shows a Coupling Facility PerfDesk with the following metrics:
sync request rate by CF structure
sync service time by CF structure
async request rate by CF structure
async service time by CF structure

Resource Measurement Facility

RMF Monitor III Data Portal


Portal
Combine strengths of Monitor III with state-of-the-art frontend
View full Reports and exploit the SORT capability

Report name

Sortable!

Scrollable!

All columns
visible!

Resizeable!

45

RMF Technical Overview

2010 IBM Corporation

In the past, the Data Portal was limited to display single metrics for selected resources
This concept has been extended: with z/OS V1.8 RMF, complete Monitor III reports can be displayed in the browser window
Thus, the strengths of the Monitor III ISPF reporting are combined with a state of the art frontend:
The window is resizable and scrollable
All report columns can be displayed (including ISPF hidden fields and the content of popup windows)
The columns can be sorted by clicking on the column header
Support of automatic refresh (GO mode)
Multiple reports can be displayed in parallel (tabbed browsing)
Print facility and Spreadsheet export

Resource Measurement Facility

RMF Performance Data API


APIs

ERBSMFI - Monitor II Data (SMF Type 79)

RMF Sysplex Data Server (SDS)


SMF

Data:
Monitor III Data:
Monitor II Data:


46

ERBDSQRY, ERBDSREC
ERB3XDRS
ERB2XDGS

RMF Provider for z/OS CIM Server

RMF Technical Overview

RMF offers a couple of powerful APIs to retrieve z/OS performance data. The exploiters can be:
Other z/OS components or systems management applications
User written applications
All RMF APIs are documented in the RMF Programmers Guide (SC33-7994)

2010 IBM Corporation

Resource Measurement Facility

Sysplex Data Server: Services

 Sysplex-wide
ERBDSQRY
ERBDSREC

access to SMF data

requests a directory of available SMF data in the sysplex


requests SMF record data in the sysplex

 Sysplex-wide

access to Monitor III data

ERB3XDRS

requests a set_of_samples of Monitor III data


does not require an ISPF and Monitor III reporter environment
provides data reduction features to transfer only the necessary data

 Sysplex-wide

access to Monitor II data

ERB2XDGS

requests Monitor II data according to the specified SMF


type 79 subtype
returns Monitor II snapshot data
provides data reduction features like ERB3XDRS

all Services are available as High-Level-Language APIs


47

RMF Technical Overview

2010 IBM Corporation

The RMF Sysplex Data Server provides a set of callable services to retrieve performance data from all RMF Monitors
Please note, that the SMF data related services are not limited to the SMF 70-78 record types: once the user has specified
any SMF record type for buffering by the Sysplex Data Server (via the SMFBUF parameter), the record images can be
obtained with the RMF services ERBDSQRY and ERBDSREQ

Resource Measurement Facility

Groovy!
Postprocessor
ZZ Session
Monitor III

Monitor II

Mintime &
Range

Interval &
Cycle Time

% Workflow

48

RMF Technical Overview

Sysplex
Data Server

Distributed
Data Server

Spreadsheet
Reporter

CIM Data
Provider

% Using / Delay

2010 IBM Corporation

Thats the freshly baked RMF expert! Probably he feels a bit more skilled now. At least he came out of his hiding-place!

Resource Measurement Facility

Information and Tools


RMF homepage: www.ibm.com/systems/z/os/zos/features/rmf/

Product information, newsletters, presentations, ...

Downloads




Spreadsheet Reporter
RMF PM Java Edition
RMF data collector for Linux

RMF email address: rmf@de.ibm.com


Documentation and news

RMF Performance Management Guide, SC33-7992

RMF Report Analysis, SC33-7991

RMF Users Guide, SC33-7990

Latest version of PDF files can be downloaded from:


http://www.ibm.com/systems/z/os/zos/bkserv/

49

RMF Technical Overview

2010 IBM Corporation

Resource Measurement Facility

RMF Redbook !!!

50

RMF Technical Overview

2010 IBM Corporation