ITSO Parallel Sysplex & High Availability Topics PDF

򔻐򗗠򙳰 ®
International Technical Support Organization
Parallel Sysplex and High Availability

2009 Update
www.ibm.com/redbooks
Frank Kyne (kyne@us.ibm.com)
Mario Bezzi
IBM ITSO - International Technical Support Organization
© 2009 IBM Corporation. All rights reserved.
ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰 ®
Welcome
My background.....
My funny accent - please remind me when I start going too fast!
Questions?? Please ask as I go along.
PLEASE complete the evaluation forms:
Especially, if you are not happy, please say WHY.
If you feel additional clarification is needed on any topics, please
indicate that as well.
Craig Ferguson
®2009 IBM Corporation. All rights reserved.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
-
2
ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰
®
Agenda
What sysplex treats does z/OS 1.11 have in store for you?
ITSO experiences with System-Managed Duplexing perfornance
Mean Time To Recovery Best Practices
Infiniband recap and performance
Misc


®
Timeline
09:00 Start
10:30 Coffee
12:00 Lunch
15:00 Afternoon Coffee
17:00 Finish

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
3
-
4
Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:
IBM has two registered trademarks for the branding of ITSO publications. These registered marks are for the text word "IBM Redbooks" and the Redbooks logo. In a nutshell,
the term Redbooks must always be used in the plural form (for both text and logo) since IBM only owns the registered mark for the plural form. Usage must follow the
guidelines below:
Using the term Redbooks in written text

Redbooks are only to be referred to in the plural form, NEVER in the singular.
For the initial reference (first occurrence), you must use "IBM Redbooks®" and include "IBM" as well as the ®. For instances thereafter you may use "Redbooks" without "IBM
preceding the word or ® following it.
Correct usage for written text :

In this IBM Redbooks® publication we will explore…..(® symbol required for 1st usage)
This Redbooks publication will show you…..(2nd usage or later - no ® or "IBM" needed)
Using the logo:
Redbooks (logo) ™
OTHER ITSO PUBLICATIONS - Marks not yet registered

Trademark registration is a lengthy process and until we are officially registered, we cannot use the ® symbol. For those terms/logos in process, we will be using the ™
symbol. In contrast to the ® symbol (placed in the lower right hand corner), the ™ symbol is placed in the upper right hand corner. Please see examples below:
TM
Redpaper ™
Redpapers ™
Redwiki ™
Redwikis ™
The following terms are trademarks of other companies:

Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or
registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.

Trademarks
The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:
eServer™ DS8000™ RACF®
pSeries® FICON® REXX™
z/OS® GDPS® RMF™
z/VM® HyperSwap™ System z™
z/VSE™ IBM® SystemPac®
z9™ IMS™ Tivoli®
BatchPipes® MVS™ VTAM®
CICS® Parallel Sysplex® WebSphere®
DB2® PAL™
DFSMShsm™ PR/SM™
DFSMSrmm™ Redbooks®
The following terms are trademarks of other companies:
Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other
countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.
CA is a trademark of Computer Associates
Other company, product, and service names may be trademarks or service marks of others.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
5
-
6
򔻐򗗠򙳰
ibm.com/redbooks z/OS 1.11 Sysplex-related enhancements
Redbooks Workshop
z/OS 1.11 Enhancements for sysplex

BCPii interface
System Status Detection Partitioning Protocol
Dynamic Failure Detection Interval calculation
Other miscellaneous XCF enhancements
Enhanced STP messages
Sysplex time-related enhancements for GDPS K system
SMF Logger improved usability
Sysplex-related Health Checks
Logger enhancements

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
7
-
8
򔻐򗗠򙳰
ibm.com/redbooks Base Control Program internal interface
Redbooks Workshop
What is BCPii?
New address space (HWIBCPII) provides authorized programs
with the ability to query, change, and perform HMC-like
functions against the System z processors on the HMC network.
Program communication from z/OS directly to HMC - no need
for TCP access from z/OS to HMC, so may help address
security concerns about exposing HMC network beyond the
machine room.
Delivered with z/OS 1.11, and rolled back to z/OS 1.10 with
APAR OA25426.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
9
-
1
0
®
BCPii interface as opposed to BCPii interface...

What is the difference between this BCPii and the BCPii
mentioned in Tivoli System Automation (TSA)?
TSA (ProcOps) allows its automation product to use one of two
transport protocols:
SNMP over an IP network
BCPii protocol (internal transport)
TSA’s BCPii implementation is similar, but is not the z/OS BCPii
and requires TSA, Netview, and Comm Server.
BCPii transport in TSA is for TSA usage only
z/OS BCPii can be used from ANY address space that has
authority and has no other software requirements.


®
BCPii functions
Exactly what can I do with the BCPii APIs?
Obtain the System z topology of the current interconnected CPCs,
Images (LPARs), and Capacity Records
Query various CPC, image (LPAR), and capacity record information
Issue commands against both the CPC and the image to perform
hardware and software-related functions (System Reset, for example)
Listen for various hardware and software events which may take place
on various CPC and images
Services are available to any address space, as long as program
is APF-authorized and has RACF access
Supports C and Assembler
Programs in USS can get notifications via Common Event Adaptor

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
1
-
1
2
®
BCPii functions
Services available
HWICONN (BCPii Connect)
HWILIST (BCPii List)
HWIQUERY (BCPii Query)
HWISET (BCPii Set) – introduced in V1R11
HWICMD (BCPii Command (only on z9 or later))
HWIEVENT (BCPii Event (for non-USS callers))
HwiBeginEventDelivery, HwiEndEventDelivery, HwiManageEvents,
HwiGetEvent (for USS callers)
HWIDISC (BCPii Disconnect)
Some of these services will be extended with additional
functions in V1R11, 1st Quarter 2010 via APAR OA29638


®
Who would use BCPii?

Vendor applications
Control center, systems management applications
In-house applications
Installations that want to manipulate System z resources
In a more automated, programmatic approach
From a z/OS environment rather than a hands-on HMC environment
Without needing to worry about an intranet/internet/internal network being
connected to the HMC network
z/OS operating system components
SFM/XCF
IBM Capacity Provisioning Manager

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
3
-
1
4
®
BCPii
In z/OS 1.11 (or 1.10 with PTF), system automatically tries to
start BCPII address space.
Successful start requires that certain setup has been carried
out:
Setup on the HMC:
Enable Cross Partition Authority for every LPAR that you want to be able to
issue or be the target of BCPii commands.
Enable SNMP and define the Community Name.
Setup in z/OS
SAF Security authorizations (in z/OS)


®
First step is to give LPARs authority to issue commands to other LPARs...
Select
CPC you
want to
set up
BCPii on
Select Single
Object Operations

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
5
-
1
6
®
You are logged

on to the SE
Select the CPC
Expand "CPC
Operational
Customization
Select "Change
LPAR Security"


®
Enable "Cross
Partition Authority"
Select Save and

Change This should update
activation profiles and
implement change on
active LPAR

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
7
-
1
8
®
Remember that
this must be
done for every
LPAR that will
exploit BCPii
This may take a

while
Press OK when
finished
Recommend verifying that

Activation Profiles were
actually updated

®
If the activation profiles were NOT updated......
You are logged

on to the HMC
Select "Customize/Delete
Activation Profiles" for the CPC
you are working on
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
9
-
2
0
®
This presents a
list of all the
Activation
(Reset and
Load) profiles
for the
selected CPC -
Select the
Reset profile
you use for this
CPC


®
This option must be

set for every LPAR
that will try to
communicate with
the HMC using BCPii
Select the
security tab for
each LPAR that
you will be using Then press Save
the BCPii from to save your
changes.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
1
-
2
2
®
Setting up BCPii - HW end

Next step is to add the SNMP definitions:
These must be added in Single Object Operations for every CPC to be
managed
SE userid must have ACSADMIN authority to be able to do this....


®
Then select
"Customize API
Logon to SE and
Settings"
select SE
Management

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
3
-
2
4
®
Select "Enable
SNMP APIs"
Then click on Add in

Community Names
section


®
Fill in exactly as
shown here.
Remember to select
Read/Write Name must be 1-16
chars, alphanumeric,
Then press OK no lower case.
Value you specify
here must match
name used in SAF
CPC profile for this
CPC
The Name value can be the

same on every CPC, or
different on every CPC. It is
NOT necessary for each CPC
to have a different Name
value if you don't wish to.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
5
-
2
6
®
Finally, click OK to
apply and save the
changes
The hardware setup for BCPii is now complete.....


®
BCPii - z/OS end

hlq.SCEERUN and hlq.SCEERUN2 must be in LNKLST.
System automatically tries to start BCPII address space at
every IPL:
Address space name is HWIBCPII.
Address space shows up in SDSF DA, but not in D A,L output.
Address space can be stopped using P HWIBCPII command:
Once the address space is stopped, no BCPII calls will be processed.
ENF signal is broadcast to let any interested parties know that
interface is stopping.
If P command doesn't work, you can use a CANCEL HWIBCPII.
Address space can be started again using S HWISTART
(HWISTART is delivered in SYS1.PROCLIB).
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
7
-
2
8
®
BCPii - Security definitions

Because BCPii provides the ability to modify vital hardware
resources, a number of security checks are performed for each
BCPii requestor. A BCPii application needs to have:
Program authority.
Authority to be able to issue BCPii commands.
Authority to the particular resource that the application is trying to
access.
A community name defined in the security product that matches the
community name specified on the SE for each CPC to which
communication is required.


®

Program authority:
Program that will be calling BCPii services must reside in an
APF-authorized library.
Issuing BCPii commands:
The profile HWI.APPLNAME.HWISERV in the FACILITY resource
class controls which applications can use BCPii services.
Anyone wishing to use BCPii must at least have READ access to this profile.
The FACILITY class must be RACLISTed.
For XCF, simply have to ensure that the XCFAS started task is defined in
RACF with the TRUSTED attribute - this is nearly always the case, but
check to be sure.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
9
-
3
0
®

A BCPii application needs to have authority to the particular
resource (CPC, Image, Capacity Record, Activation Profile) that
it is trying to access (This is IN ADDITION to having access to
the HWISERV FACILITY profile).
Profile names are:
CPC: HWI.TARGET.netid.nau
Image: HWI.TARGET.netid.nau.imagename
Capacity Record: HWI.CAPREC.netid.nau.caprec
Activation Profile: HWI.TARGET.netid.nau
netid.nau is the 3-17 character SNA name for CPC (defined when you
first define the SE to the HMC)
Level of access that is required depends on what you are trying
to do - See Callable Services manual for details

®

When defining the CPC profiles, APPLDATA must match the community
name you specified on the SE:
RDEFINE FACILITY HWI.TARGET.USIBMSC.SCZP201 UACC(NONE) APPLDATA(‘BCPII’)
You will need

one of these for
EACH CPC that
will be managed
using BCPii

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
3
1
-
3
2
®
BCPii Prerequisites
Software:
z/OS 1.11 (included in the base)
z/OS 1.10 with APAR OA25426
Hardware:
The program issuing the BCPii calls must be running on any CPC
supported by z/OS 1.11 (z900 or later)
It is always wise to keep CPCs (even old ones) at current microcode levels
The HWICMD function can only be used against z9 or later with the
following microcode levels:
z9: G40965.133
z10: F85906.116


®
BCPii further information

z/OS 1.11 MVS Programming: Callable Services for High-Level
Languages:
Primary BCPii documentation including installation instructions and
BCPii API documentation.
z/OS 1.11 MVS System Commands:
START HWISTART and STOP HWIBCPII commands.
z/OS 1.11 MVS Diagnosis: Tools and Service Aids:
BCPii’s CTRACE documentation.
z/OS MVS Programming: Authorized Assembler Services
Reference, Volume 2 (EDT-IXG):
BCPii’s ENF68 documentation.
Various SHARE presentations - see www.share.org
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
3
3
-
3
4
򔻐򗗠򙳰
ibm.com/redbooks System Status Detection Partitioning Protocol
Redbooks Workshop
System Failure Handling in a sysplex

The design mode of operation for a sysplex is that all resources
should be accessible from all members of the sysplex and any
work should be able to run anywhere in the sysplex.
This means that it should be possible to maintain application
availability even if one or more of the systems in the plex are
down.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
3
5
-
3
6
The downside is that if a MVSA

Job AA
system dies: DSN=A EXCL
DSN=B EXCL
It is probably holding
resources that will be
needed by another member
of the sysplex. Waiting
MVSB MVSD
It cannot release those
Job BB Job DD
resources because it is DSN=B EXCL DSN=D EXCL
DSN=E EXCL
dead.
DSN=C EXCL
Job DD2 Waiting

Waiting
The longer a dead system Waiting MVSC
DSN=E SHR
remains in the sysplex Job CC

(holding resources), the DSN=C EXCL
DSN=D EXCL
larger is the impact on

other systems.

To address this situation, the dead system should be Reset (or
Fenced) and then partitioned from the sysplex:
Resetting the system (or Fencing it) ensures that it can't do any more
I/O (thereby protecting data integrity) and frees up any RESERVES
held by that LPAR.
Partitioning the system frees up any ENQs held by that system,
thereby letting the other members of the sysplex proceed with their
work.
So, it seems obvious that the best way to address this is to
partition a dead system as quickly as possible....
The challenge is - how does z/OS know if another member of the
sysplex is dead?

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
3
7
-
3
8
®

In order to let its peers know its status, each member of a
sysplex updates its heartbeat in the Sysplex Couple Data Set
every few seconds.
So, if a system hasn't updated its heartbeat in the last 10 seconds,
does that mean it is dead?
What if the system has some transient problem that is stopping it

from updating its heartbeat - maybe the operator pressed STOP for
that LPAR, or it is stuck in a spin loop? In these cases, the system may
be able to recover if we give it enough time.


®

If the system doesn't resume updating its timestamp after some
amount of time, XCF would eventually prompt the operator with
an IXC402D WTOR, asking the operator to reply DOWN when
the system in question had been RESET.
But what happens if the operator is on his coffee break?
And what happens if he replies DOWN without doing the Reset?
And how long should we give the system to see if it can recover from
whatever is stopping it from updating its heartbeat?

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
3
9
-
4
0
®

Of the various things that can stop an LPAR from updating its
heartbeat in the CDS, the one with a finite duration is spin loop
recovery. Therefore, the amount of time that a system should
be allowed to update its heartbeat in the CDS depends largely
on:
The LPAR configuration. LPARs with shared engines need to be given
more time to recover from spin loops than those with dedicated
engines.
The spin recovery actions that the installation has specified and
activated using the EXSPATxx member.
The amount of time (prior to z/OS 1.11) that a system IS
allowed between heartbeats is controlled by the Failure
Detection Interval specified in the COUPLExx member.

®

The Failure Detection Interval (prior to z/OS 1.11) defaults to
either 25 seconds (LPAR with dedicated CPs) or 85 seconds
(LPAR with shared CPs).
OR... it can be overridden in the COUPLExx member

/********************************************************************/
/* MEMBER = COUPLE00 */
/* */
/* DESCRIPTION = THIS PARMLIB MEMBER IS ACCESSED THROUGH THE */
/* IEASYSXX MEMBER BY SPECIFYING COUPLE=00 */
/********************************************************************/
COUPLE SYSPLEX(&SYSPLEX.)
PCOUPLE(SYS1.XCF.MAXSYS12.CDS01)
ACOUPLE(SYS1.XCF.MAXSYS12.CDS02) This changes in
z/OS 1.11, but more
INTERVAL(85)
OPNOTIFY(85)
CLEANUP(15)
RETRY(10)
VMCPUIDTOLERATION(YES) about that later....
CFRMOWNEDCFPROMPT(YES)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
4
1
-
4
2
®

But, coming back to our operator - what happens if no one
replies to the IXC402D WTOR?
NOTHING
The dead or sick system doesn't get partitioned from the
sysplex.
And the other members of the sysplex will slowly grind to a halt
as they try to serialize resources held by that system.


®

To help get the sick/dead system out of the way as quickly as
possible, IBM introduced the Sysplex Failure Management
component of XCF.
This can (under installation control) automatically partition a
system from the sysplex if:
The Failure Detection Interval has been reached AND no heartbeat
has been received AND the apparently dead system is not sending any
XCF signals.
To try to encourage customers to use SFM, health checks were
provided to ensure that there is an active SFM policy and that
the policy specified the ISOLATETIME option.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
4
3
-
4
4
®

But what happens if the system is in a wait state (it is already
dead)? Do we still need to wait for the whole FDI before we
take some action?
Prior to z/OS 1.11 XCF had no way to tell the difference
between a system that could not recover (non-restartable wait
state, for example) and one that might be able to recover (spin
loop). The indicator of the health of a system was the
heartbeat in the CDS and the absence of XCF signals.
Therefore, XCF/SFM had to wait for the full Failure Detection
Interval in every case.


®
An aside......
Is V XCF,sysname,OFFLINE the same as SFM ISOLATE
actions?
NO!
Issuing a Vary XCF command sends a signal to the named system
telling it to shutdown and place itself in a wait state.
An SFM-initiated ISOLATE sends a signal to the processor the target
system is running on, telling it to isolate the I/O subsystem of the
target LPAR.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
4
5
-
4
6
®

Before we discuss z/OS 1.11, let's summarize how XCF handled
a dead OR spinning system prior to z/OS 1.11:
If SFM was NOT active, the operator would eventually be prompted with
message IXC402D, asking him to RESET the LPAR, then reply DOWN.
If SFM WAS active and ISOLATETIME was specified (as recommended),
SFM would (eventually) attempt to automatically Fence the problem
system and partition it out of the sysplex.
But this required a Coupling Facility - without the CF (in a base sysplex, for
example) there is no ability to fence a system.
If the operator spotted messages indicating that a system appeared
non-responsive, they could check the system status on the HMC and take
manual action if the system was in fact dead.
z/OS 1.11 introduces some fundamental changes to this
philosophy.....

®

Change number 1 in z/OS 1.11:
Starting with z/OS 1.11, the default action if there is no active SFM
policy HAS CHANGED:
With 1.11, if SFM is NOT active, the default SSUM action is now ISOLATE
instead of PROMPT.
With 1.11, if SFM is active with ISOLATETIME specified, the desired
action of ISOLATE will be taken.
The only way to get a 1.11 system to indicate that PROMPT is the desired
action is to have an active SFM policy, and specify PROMPT in the policy.
But this is not the recommended course of action.
THIS CHANGE IS IRRESPECTIVE OF SSD (change # 2)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
4
7
-
4
8
®

Even though the D XCF,C command only shows two possible
actions for SSUM conditions (ISOLATE or PROMPT), in reality
the Sysplex CDS can contain 3 actions:
ISOLATE, if SFM is active with ISOLATE
PROMPT, if SFM is active with PROMPT
"Take The Default Action", if SFM is not active.
The difference is that for releases prior to 1.11, the "default
action" means PROMPT. For 1.11, the "default action" is
ISOLATE.
Important: z/OS 1.11 understands that "the default action" for a
pre-R11 system is PROMPT, and ISOLATE for an R11 or higher system.
Pre-z/OS 1.11 does NOT understand that "the default action" for an
R11 system is ISOLATE.

®

Change number 2:
New in z/OS 1.11 is the elegantly-named "System Status
Detection Partitioning Protocol".
What is the System Status Detection Partitioning Protocol
(SSD)?
An enhancement to failed-system handling designed to attempt

to partition a failed system from the sysplex in a more timely
way and with improved data integrity.
SSD achieves this by exploiting the new BCPii support to
communicate with the SE about the status of an LPAR.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
4
9
-
5
0
®

When a z/OS 1.11 system is IPLed using a new-format Sysplex
CDS, it writes new information about itself into the CDS. This
information includes information it gets via BCPii:
The network name of the CPC it is running on (netid.nau).
The name of the LPAR it resides in.
An IPL Token.
Both the hardware and the software know the IPL Token:
The IPL token is valid for the life of the IPL, as long as the system is still
functioning.
If the LPAR is RESET, the IPL Token in the hardware will change.
If the LPAR waitstates (non-restartable), the IPL Token in the hardware will change.
If the LPAR is IPLed, the IPL token will change.
All of this information is available to the other members of the

sysplex via the Sysplex CDS and the BCPii.

®

D XCF,C
IXC357I 14.51.34 DISPLAY XCF 275
SYSTEM SC80 DATA
INTERVAL OPNOTIFY MAXMSG CLEANUP RETRY CLASSLEN
165 168 2000 15 10 956
SSUM ACTION SSUM INTERVAL SSUM LIMIT WEIGHT MEMSTALLTIME

ISOLATE 0 NONE 1 NO
SYSTEM STATUS DETECTION PARTITIONING PROTOCOL ELIGIBILITY:

SYSTEM CAN TARGET OTHER SYSTEMS.
SYSTEM IS ELIGIBLE TO BE TARGETED BY OTHER SYSTEMS.
SYSTEM NODE DESCRIPTOR: 002097.IBM.02.00000001DE50

PARTITION: 01 CPCID: 00
SYSTEM IDENTIFIER: DE502097 010000C9
Obtained via BCPii (if

NETWORK ADDRESS: USIBMSC.SCZP201
SSD is active)
PARTITION IMAGE NAME: A01
IPL TOKEN: C4B6E90D BCA7040E

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
5
1
-
5
2
®

System Status Detection Partioning Protocol also introduces
other changes, enabled by the new ability to communicate with
the hardware via the BCPii.
Previously, the best way for a system to determine if another
member of the sysplex was dead was to monitor its heartbeat in
the Sysplex CDS and wait for the FDI to expire.
If the BCPii is active and the Sysplex CDS correctly formatted,
XCF is now able to go out to the hardware as soon as an LPAR
stops updating its heartbeat and query the status of the LPAR.
If the system is in a non-restartable wait state (as indicated by
its IPL Token) or has been System Reset, XCF will immediately
Reset it and partition it out of the sysplex.


®

Why is this so great??
We no longer have to wait for the entire FDI to partition a wait-stated
system. This means that all the other members of the sysplex should be
able to resume normal processing sooner.
In certain situations it may be advantageous to give the system more
time to recover from a spin loop:
However changing the spin recovery options (in EXSPAT) so that recovery can
run for longer means that the FDI should be adjusted accordingly.
This was OK if the system was spinning (because maybe the extra time will
help the system recover), but what if the system was in a wait state? We
would now have to wait even longer to partition the system from the sysplex.
So some customers were torn between having "better" spin recovery options,
and trying to partition dead systems as quickly as possible.
System Status Detection Partitioning Protocol lets you have the best of both
worlds - longer spin recovery, but minimal time to partition dead systems.
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
5
3
-
5
4
®

And even MORE great things...
Prior to z/OS 1.11, an operator could reply DOWN to the IXC402D
message even if the target LPAR had not actually been reset.
As soon as the DOWN was received, partitioning actions would start -
including releasing ENQs held by the "dead" system.
But what if that system only appeared to be dead? If the operator did not
Reset the LPAR, it was possible for the system to come back to life and
starting processing data that it thought it had serialization on - thereby
creating an integrity exposure.


®

With SSD, XCF can query the status of the LPAR to ensure that
it has really been reset - this was not possible previously.
*0016 IXC402D SY2 LAST OPERATIVE AT 11:13:46. REPLY DOWN AFTER SYSTEM
RESET, OR INTERVAL=SSSSS TO SET A REPROMPT TIME.
R 16,DOWN Even though they have
not done the System
IEE600I REPLY TO 0016 IS;DOWN
IXC114I LOGICAL PARTITION REMOTE STATUS INFORMATION 197 Reset yet

SYSTEM NAME: SY2
SYSTEM NUMBER: 02000006
CPC: D13ID32.SY2 STATUS: OPERATING OPERSTAT: 00000001
IMAGE NAME: SY2 STATUS: OPERATING OPERSTAT: 00000001 From Sysplex CDS
STORED IPL TOKEN: C41EAFBA 46653F0E
RETURNED IPL TOKEN: C41EAFBA 46653F0E
DIAG INFO: N/A
From Hardware
(via BCPii)
IXC208I THE RESPONSE TO MESSAGE IXC402D IS INCORRECT: DOWN REPLY ENTERED
WITHOUT SYSTEM RESET

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
5
5
-
5
6
®

R 34,DOWN
Status is now no
IEE600I REPLY TO 0034 IS;DOWN
IXC114I LOGICAL PARTITION REMOTE STATUS INFORMATION longer OPERATING,
SYSTEM NAME: SY2 so partitioning can
SYSTEM NUMBER: 02000009
CPC: D13ID32.SY2 STATUS: OPERATING OPERSTAT: 00000001 proceed safely
IMAGE NAME: SY2 STATUS: NOT ACTIVATED OPERSTAT: 00000008
STORED IPL TOKEN: C4276E16 DE0BB90F
IPL Token has been

RETURNED IPL TOKEN: 00000000 00000000
DIAG INFO: N/A
reset, so no longer
matches token in
IXC101I SYSPLEX PARTITIONING IN PROGRESS . . .
IXC113I BCPII CONNECTION TO SYSTEM SY2 RELEASED. . . Sysplex CDS

IXC105I SYSPLEX PARTITIONING HAS COMPLETED FOR. . .
More than likely, when you do the System Reset, SSD will see this
very quickly and partition the system WITHOUT issuing another
IXC402D. When SSD is active, XCF will ALWAYS immediately
partition a wait-stated or System Reset system.

®

OK, this sounds good to me - what do I need to do to get this?
The systems that will drive the System Status Detection Partitioning
Protocol processing, or be the target of such processing, MUST be
running on z10 EC GA2 or z10 BC GA1 or later.
BCPii must be configured and functioning.
XCFAS must be defined as TRUSTED to RACF or must have access to
the required BCPii SAF profiles.
Because the interaction with BCPii requires information that only z/OS
1.11 systems place in the Sysplex CDS, only z/OS 1.11 systems can have
the new protocol used to determine their status.
Because the new System Status Detection Partitioning Protocol support
is only in z/OS 1.11 and subsequent releases, only 1.11 or later systems
can use this new protocol to determine the status of another system.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
5
7
-
5
8
®

Anything else?
Because new information is stored in the Sysplex CDS, you need a new
format CDS:
New in this release is the ability of back-level systems to be able to use an
up-level CDS! Requires toleration APAR OA26037 on the pre-1.11 systems.
However this means that you can potentially start to benefit from this new
protocol as soon as at least 2 members of the sysplex are running 1.11 - you
don't have to wait for the whole sysplex to move to 1.11.
You can turn the use of SSD on or off at the system level using the
SETXCF FUNCTIONS capability:
Default is ENABLED - this is the recommended setting.
If you DISABLE SSD on one system using SETXCF
FUNCTIONS,DISABLE=SYSTEMSTATDETECT, that system cannot be the
target of any BCPii-related actions and will not use BCPii to initiate actions
against any other systems.

®

OK, I like the idea and I have all the pre-reqs, how do I enable
this?
Make sure that BCPii is set up (as per earlier section).
Format 2 new Sysplex CDSs using the SSTATDET keyword:
//DEFCOUPL JOB (999,POK),'SYSPLEX CDS',NOTIFY=&SYSUID,
// CLASS=A,MSGCLASS=T,TIME=1439,
// REGION=0M,MSGLEVEL=(1,1)
/*JOBPARM SYSAFF=*
//STEP1 EXEC PGM=IXCL1DSU
//SYSPRINT DD SYSOUT=*
//SYSIN DD *
DEFINEDS SYSPLEX(WTSCPLX8)
MAXSYSTEM(4)
DSN(SYS1.XCF.CDS02) VOLSER(BH8CD1) CATALOG
DATA TYPE(SYSPLEX)
ITEM NAME(GRS) NUMBER(1)
ITEM NAME(GROUP) NUMBER(200)
ITEM NAME(MEMBER) NUMBER(203)
ITEM NAME(SSTATDET) NUMBER(1)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
5
9
-
6
0
®

Enabling SSD (cont)...
Use the SETXCF COUPLE,ACOUPLE=dsn and SETXCF
COUPLE,PSWITCH commands to roll the new CDSs into production.
Note that after you activate a new CDS formatted for SSD, it may take a
few seconds before you see:
IXC103I SYSTEM IDENTIFICATION INFORMATION 269
CONNECTION STATUS: CONNECTED
SYSTEM NAME: SC80
SYSTEM NUMBER: 010000C9
IMAGE NAME: A01
NODE DESCRIPTOR: 002097.IBM.02.00000001DE50
PARTITION NUMBER: 01
CPC ID: 00
IXC104I SYSTEM STATUS DETECTION PARTITIONING PROTOCOL ELIGIBILITY: 270


®

How to find out if your Sysplex CDS is formatted to support
SSD:
D XCF,C,TYPE=SYSPLEX
SYSPLEX COUPLE DATA SETS
PRIMARY DSN: SYS1.XCF.CDS03
VOLSER: BH8CD2 DEVN: D24E
FORMAT TOD MAXSYSTEM MAXGROUP(PEAK) MAXMEMBER(PEAK)
09/02/2009 14:43:06 4 200 (46) 203 (7)
ADDITIONAL INFORMATION:
ALL TYPES OF COUPLE DATA SETS ARE SUPPORTED
GRS STAR MODE IS SUPPORTED
SYSTEM STATUS DETECTION PROTOCOL IS SUPPORTED

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
6
1
-
6
2
®

How do I know if it is working?
Tells us that
Watch for messages - IXC or HWI. Sysplex CDS is
Check output from D XCF,C command. formatted for SSD,
that BCPii is
D XCF,C working, and that
IXC357I 14.51.34 DISPLAY XCF 275 SSD has not been
SYSTEM SC80 DATA
INTERVAL OPNOTIFY MAXMSG CLEANUP RETRY CLASSLEN disabled using the
165 168 2000 15 10 956 FUNCTIONS
SSUM ACTION SSUM INTERVAL SSUM LIMIT WEIGHT MEMSTALLTIME command
ISOLATE 0 NONE 1 NO
.....



®

What other messages might I see?
Some problem with the BCPii
SYSTEM STATUS DETECTION PARTITIONING PROTOCOL ELIGIBILITY: setup (check if HWIBCPII
SYSTEM CANNOT TARGET OTHER SYSTEMS. address space is started, or
REASON: BCPII SERVICES NOT AVAILABLE
SYSTEM IS NOT ELIGIBLE TO BE TARGETED BY OTHER SYSTEMS. for HWIxxxx messages)
REASON: BCPII SERVICES NOT AVAILABLE
SSD has been turned off

using the SETXCF
SYSTEM CANNOT TARGET OTHER SYSTEMS.
REASON: NOT ENABLED BY INSTALLATION FUNCTIONS command
SYSTEM IS NOT ELIGIBLE TO BE TARGETED BY OTHER SYSTEMS.
REASON: NOT ENABLED BY INSTALLATION
CDS not formatted

SYSTEM CANNOT TARGET OTHER SYSTEMS. correctly
REASON: SYSPLEX COUPLE DATA SET NOT FORMATTED FOR THE PROTOCOL

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
6
3
-
6
4
®

How do I know if it is working (cont)?
All of this comes
D XCF,C from the BCPii
..... interface
PARTITION IMAGE NAME: A01

Whether SSD was
COUPLEXX PARMLIB MEMBER USED AT IPL: COUPLE00 disabled with the
OPTIONAL FUNCTION STATUS: FUNCTIONS
FUNCTION NAME STATUS DEFAULT command (default is
DUPLEXCF16 DISABLED DISABLED
SYSSTATDETECT ENABLED ENABLED ENABLED)
USERINTERVAL DISABLED DISABLED


®

Considering that SSD and non-SSD systems can coexist in the
same sysplex, what does that mean for me if the system is
still not-dead?
"Sick" system Pre-R11 Pre-R11 Pre-R11 R11+ R11+ R11+ SFM
SFM Active SFM Active No SFM No SFM SFM Active Active
ISOLATE PROMPT ISOLATE PROMPT
Partitioning system
Pre-R11 ISOLATE IXC402D IXC402D IXC402D ISOLATE IXC402D
R11 ISOLATE IXC402D IXC402D ISOLATE ISOLATE IXC402D
Note that if an R11 system can determine that another R11 system is
dead (via the BCPii), none of the above applies - the detecting system will
immediately partition the dead system from the sysplex.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
6
5
-
6
6
®

Does this mean that I don't need to use SFM any more?
NO! SFM has other capabilities that you will lose if you turn
off SFM:
System weights can only be assigned via an SFM policy.
MEMSTALLTIME (delivered in z/OS 1.8) identifies and potentially
terminates stalled XCF exploiters that are causing sympathy sickness.
SSUMLIMIT (delivered in z/OS 1.9) identifies systems that are not
updating their heartbeat but also are not dead, and provides the option
to have them automatically partitioned after x amount of time.


®

Any other considerations?
Do NOT confuse SSD with AutoIPL (introduced in z/OS R10).
AutoIPL can IPL a system that enters selected disabled wait states -
IT ADDRESSES THE FAILED SYSTEM.
SSD addresses all the OTHER MEMBERS OF THE SYSPLEX - it gets
the other members of the sysplex to clean up their information about
the failed system.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
6
7
-
6
8
®

Any other considerations?
If you move to z/OS 1.11 but don't or can't activate BCPii, your Failure
Detection Interval will be longer than it was previously (probably 165
seconds vs 85 seconds), but you won't have the benefit of the faster
partitioning for dead systems that BCPii enables.
In this case, you might want to use the FUNCTIONS USERINTERVAL
switch to reset the FDI back to your current value (but remember that the
recommended value really should have been 165 seconds all along) until you
enable BCPii.


®

Summary:
System Status Detection Partitioning Protocol is a significant step
forward. This is the most fundamental change to handling of system
failures since the introduction of SFM.
XCF WILL automatically partition an SSD-enabled system from the
sysplex if it is really dead, regardless of the SSUM ACTION value.
Builds on concepts we are all familiar with.
But don't confuse SSD with the change of the SSUM ACTION default in
z/OS 1.11. The default changes to ISOLATE regardless of whether SSD is
enabled or not.
Easy to implement.
You can start to enable it as soon as your first z10 z/OS system moves
to z/OS 1.11 - no need to wait for the whole sysplex to be upgraded.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
6
9
-
7
0
򔻐򗗠򙳰
ibm.com/redbooks XCF Failure Detection Interval Consistency
Redbooks Workshop
XCF FDI Consistency

The XCF Failure Detection Interval is intended to reflect the
longest amount of time that a system is likely to run without
updating its heartbeat in the Sysplex Couple Data Set, while still
being able to resume normal operations.
The most likely scenario for a system to run for a long time
without being able to update its heartbeat in the Couple Data
Set is if it is waiting for a spin lock to be released.
What controls how long the system will stay in this state, and
what happens if the lock is not released?

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
7
1
-
7
2
®
XCF FDI Consistency Then CP2 tries to

get the lock, also
What is a spin loop? loops CP3
continues
with other
work
CP0 CP1 CP2 CP3
CP0 is holding
Holding
a lock while it lock Normal
Request work
does lock
something request Request The EXSPAT
CP1 tries to get request lock member controls
the lock, and request request what actions are
goes into a request taken and for
disabled loop for request how long
SPINTIME
trying to get it

®
XCF FDI Consistency

The possible options in EXSPATxx are:
SPIN - Go into a disabled loop for up to SPINTIME seconds, trying to
obtain the lock.
ABEND - Abend the work that is holding the processor (or lock) and
allow recovery routines to get control.
TERM - Terminate the work holding the processor and do NOT allow
the recovery routines to get control.
ACR - Alternate CPU Recovery - the processor holding the lock is taken
offline.
OPER - Prompt the operator.
The default actions are:
ABEND,TERM,ACR
The system automatically spins for up to 2 * SPINTIME before invoking the
actions specified in EXSPATxx.
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
7
3
-
7
4
®
XCF FDI Consistency
Worst case timeline for a spin loop (using default EXSPAT

options of ABEND,TERM,ACR):
00:00:00 Program A gets lock and starts looping
00:00:25 Program B tries to get lock and starts spinning, trying to obtain the lock.
Program B will be allowed to spin for up to the SPINTIME value.
+40 System detects situation and kicks off spin recovery processing with 1 SPIN
action (that is, keep trying for up to another 40 seconds).* System creates
S071-10 LOGREC record and message IEE178I.
+80 Lock still not released. Spin recovery tries to ABEND Program A.
. Program A is abended, but recovery routine does not release the lock.
System creates a S071-20 LOGREC record and message IEE178I.
+120 Lock still not released so we take the next action - TERM. Program is
Terminated without recovery. System creates a S071-30 LOGREC record
and message IEE178I.
+160 If lock is STILL not released, CP holding the lock taken offline.
* Regardless of what is specified in the EXSPATxx member, the first action in spin recovery will always be to spin for up to
SPINTIME more seconds in the hope that the lock will be released. This is referred to as the "free spin".

®
XCF FDI Consistency

Prior to z/OS 1.11, the default Failure Detection Interval was:
2 * DEFAULT SPINTIME + 5
The default SPINTIME depends on the config - 40 seconds for shared
engines, 10 seconds for dedicated.
However if you changed the EXSPAT member to have more
actions or a different SPINTIME, the Failure Detection Interval
would not be adjusted automatically.
This could result in a system being partitioned out of the
sysplex while it is still trying to recover from a spin loop (if the
FDI is not high enough) OR a dead system being left in the
sysplex longer than necessary (if the FDI was too high).

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
7
5
-
7
6
XCF FDI Consistency

In z/OS 1.11, the Failure Detection Interval may automatically
be adjusted to reflect the spin recovery actions and spin times
specified in the EXSPATxx member.
Note that the default EXSPAT actions will be used unless you
explicitly issue a SET EXS=xx command. The EXSPAT00 member is
NOT automatically processed, and there is no IEASYSxx parameter to
point at a particular EXSPATxx member.

XCF FDI Consistency

But first, there are some new terms in 1.11:
Prior to z/OS 1.11, this D XCF,C

was either the default SYSTEM SC80 DATA
value, or the 165 168 2000 15 10 956
user-specified SSUM ACTION SSUM INTERVAL SSUM LIMIT WEIGHT MEMSTALLTIME

INTERVAL value from ISOLATE 0 NONE 1 NO
COUPLExx if DEFAULT USER INTERVAL: 165
specified. DERIVED SPIN INTERVAL: 165 The value

DEFAULT USER OPNOTIFY: + 3
In 1.11 and later, displayed here
(and the wording)
MAX SUPPORTED CFLEVEL: 16
INTERVAL is the
greater of DERIVED depend on a
SPIN INTERVAL or number of things
INTERVAL value from
COUPLExx or SETXCF This is calculated using
command. the spin recovery actions
and the SPINTIME value
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
7
7
-
7
8
®
XCF FDI Consistency

D XCF,C
SYSTEM SC80 DATA
INTERVAL OPNOTIFY MAXMSG CLEANUP RETRY CLASSLEN This means that an
INTERVAL value was NOT
165 168 2000 15 10 956
specified in COUPLExx or
ISOLATE 0 NONE 1 NO
DEFAULT USER INTERVAL: 165

via a SETXCF command
DERIVED SPIN INTERVAL: 165
D XCF,C
IXC357I 14.03.19 DISPLAY XCF 047 This means that an INTERVAL
SYSTEM SC80 DATA
value WAS specified via a
105 108 2000 15 10 956 SETXCF command.
SSUM ACTION SSUM INTERVAL SSUM LIMIT WEIGHT MEMSTALLTIME If COUPLExx contains INTERVAL
ISOLATE 0 NONE 1 NO
keyword, this will say
SETXCF USER INTERVAL: 85 PARMLIB USER INTERVAL


®
XCF FDI Consistency

Does this mean that the system will override my override??
It depends...
By default, the system will use what it deems to be the safest value.
If you insist that it use your override regardless of whether the
system considers that a "good" value or not, you can force it to behave
as it did prior to z/OS 1.11.
SETXCF FUNCTIONS,ENABLE=USERINTERVAL (NOT the
default) tells the system to always use whatever value you
specify on the INTERVAL keyword.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
7
9
-
8
0
®
XCF FDI Consistency

Let's see an example:
SETXCF COUPLE,INTERVAL=85
IXC470I SYSTEM SC80 EFFECTIVE VALUES: INTERVAL=165 OPNOTIFY=168 044 Operator issued command
SETXCF USER INTERVAL: 85
DERIVED SPIN INTERVAL: 165 to set Failure Detection
Interval to 85 seconds
COMPUTED FOR: SETXCF COUPLE,INTERVAL
IXC309I SETXCF COUPLE,INTERVAL REQUEST WAS ACCEPTED
D XCF,C
SYSTEM SC80 DATA
165 168 2000 15 10 956

ISOLATE 0 NONE 1 NO
Based on the spin
SETXCF USER INTERVAL:
DERIVED SPIN INTERVAL:
85
165
recovery options, XCF has
DEFAULT USER OPNOTIFY: + 3 determined that the
Failure Detection Interval
should be 165 seconds


®
XCF FDI Consistency

Let's see an example:
Operator increased FDI
IXC470I SYSTEM SC80 EFFECTIVE VALUES: INTERVAL=185 OPNOTIFY=188 049 above value XCF derived
DERIVED SPIN INTERVAL: 165 from spin recovery options
D XCF,C
SYSTEM SC80 DATA
185 188 2000 15 10 956
The user requested a

ISOLATE 0 NONE 1 NO

185
165
LARGER value, so XCF
honors that larger value
Note that OPNOTIFY is
using offset rather than
absolute value
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
8
1
-
8
2
®
XCF FDI Consistency

Let's go back to INTERVAL=85:
Set INTERVAL to 85
IXC470I SYSTEM SC80 EFFECTIVE VALUES: INTERVAL=165 OPNOTIFY=168 210
D XCF,C
SYSTEM SC80 DATA
165 168 2000 15 10 956
Actual INTERVAL is
PROMPT 168 N/A 1 NO

85
165
larger of SETXCF USER
INTERVAL and DERIVED
SPIN INTERVAL


®
XCF FDI Consistency

Let's go back to INTERVAL=85:
Tell THIS system to

SETXCF FUNCTIONS,ENABLE=USERINTERVAL
IXC373I XCF / XES OPTIONAL FUNCTIONS ENABLED: 224
USERINTERVAL
IXC470I SYSTEM SC80 EFFECTIVE VALUES: INTERVAL=85 OPNOTIFY=88 225 use your override.
COMPUTED FOR: SETXCF FUNCTIONS
D XCF,C
SYSTEM SC80 DATA
85 88 2000 15 10 956
Actual INTERVAL is now

PROMPT 88 N/A 1 NO

the SETXCF USER
DEFAULT USER OPNOTIFY: +
165
3
INTERVAL value

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
8
3
-
8
4
XCF FDI Consistency

Summary:
Combined with the System Status Detection Partitioning Protocol, this
enhancement should result in improved availability:
Truly dead systems will be partitioned from the sysplex faster.
You can give the system longer to recover from a spin loop (that is, specify a
larger FDI value) potentially without the negative impact that previously had
on partitioning responsiveness.
This function will be turned on by default - no actions required on your
behalf.
The time allowed for recovery from a spin loop may double compared to
pre-R11 (depending on your current EXSPAT and FDI values) - so you
may need to take this into account for your operational procedures.

򔻐򗗠򙳰
ibm.com/redbooks Other Miscellaneous XCF changes in z/OS 1.11
Redbooks Workshop

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
8
5
-
8
6
®
Other XCF enhancements

z/OS 1.11 introduces a refinement to the OPNOTIFY definition.
Prior to 1.11, the OPNOTIFY value included the FDI. So if FDI was
100 and you wanted to tell the operator 5 seconds later, you would set
OPNOTIFY to 105. If you adjust FDI, you also needed to adjust
OPNOTIFY.
In z/OS 1.11, you have the option of specifying OPNOTIFY as an
offest. So, in this example, instead of OPNOTIFY(105), you could say
OPNOTIFY(+5). Then if you change the FDI value, there is no need to
update OPNOTIFY.
In the example earlier, you may have noticed how the OPNOTIFY value
gets adjusted every time the INTERVAL value changes, but without
having to make any explicit changes.


®

Improved support for large numbers of subchannels.....
Back in the early days of sysplex (when CF links only had 2
subchannels each) IBM used to recommend that customers
configure 2 links to each CF "for availability".
Today, we have customers with 8 links to each CF (each with 7
subchannels) for performance reasons:
Very large sysplexes (100K MIPS).
Large distances between z/OS and the CF.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
8
7
-
8
8
®

In z/OS 1.11 (and with APAR OA26033 back to z/OS 1.7), the
recovery process has been optimized to better handle
configurations with a large number of subchannels to a given CF.
The output from D CF and D M= commands has also been
enhanced to make them more user-friendly.


®
D CF command BEFORE OA26033

D CF,CFNM=FACIL03
IXL150I 22.00.02 DISPLAY CF 210
COUPLING FACILITY 002097.IBM.02.00000001DE50
PARTITION: 0E CPCID: 00
CONTROL UNIT ID: FFE4
NAMED FACIL03
SENDER PATH PHYSICAL LOGICAL CHANNEL TYPE
B4 / 0014 OFFLINE ONLINE CBP
B5 / 0015 OFFLINE ONLINE CBP
C4 OFFLINE ONLINE ICP
C5 OFFLINE ONLINE ICP
D8 ONLINE ONLINE ICP
COUPLING FACILITY SUBCHANNEL STATUS

TOTAL: 56 IN USE: 14 NOT USING: 0 NOT USABLE: 42
DEVICE SUBCHANNEL STATUS
FCEC 3E08 OPERATIONAL
FCED 3E09 OPERATIONAL
FCEE 3E0A OPERATIONAL
FCEF 3E0B OPERATIONAL
FCF0 3E0C OPERATIONAL
FCF1 3E0D OPERATIONAL
FCF2 3E0E OPERATIONAL
FCF3 3E0F OPERATIONAL
FCF4 3E10 OPERATIONAL
FCF8
IBM ITSO - International Technical Support Organization 3E14 OPERATIONAL
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
8
9
-
9
0
®
D CF command AFTER OA26033

D CF,CFNM=FACIL03
IXL150I 22.00.16 DISPLAY CF 887
COUPLING FACILITY 002097.IBM.02.00000001DE50
PARTITION: 0E CPCID: 00
CONTROL UNIT ID: FFE4
NAMED FACIL03
.....
COUPLING FACILITY SPACE UTILIZATION
SENDER PATH PHYSICAL LOGICAL CHANNEL TYPE
B4 / 0014 PATH NOT AVAILABLE ONLINE CBP
B5 / 0015 PATH NOT AVAILABLE ONLINE CBP
C4 PATH NOT AVAILABLE ONLINE ICP
C5 PATH NOT AVAILABLE ONLINE ICP
COUPLING FACILITY SUBCHANNEL STATUS

TOTAL: 56 IN USE: 14 NOT USING: 0 NOT USABLE: 42
OPERATIONAL DEVICES / SUBCHANNELS:
FCEC / 3E08 FCED / 3E09 FCEE / 3E0A FCEF / 3E0B
FCF0 / 3E0C FCF1 / 3E0D FCF2 / 3E0E FCF3 / 3E0F
FCF4 / 3E10 FCF5 / 3E11 FCF6 / 3E12 FCF7 / 3E13
FCF8 / 3E14 FCF9 / 3E15 FDC5 / 3E16 FDC6 / 3E17
FDC7 / 3E18 FDC8 / 3E19 FDC9 / 3E1A FDCA / 3E1B
FDCB / 3E1C FDDA / 3E1D FDDB / 3E1E FDDC / 3E1F
FDDD / 3E20 FDDE / 3E21 FDDF / 3E22 FDE0 / 3E23


®

APAR OA26856 (for z/OS R9/10/11) delivers improved
recovery/handling for XCF signalling failures.
Prior to this APAR, XCF could not determine which “in doubt”
signals needed to be resent until the path recovery process
completed.
Sometimes the path recovery process takes a long time, meaning
that undelvered signals experience delay.
Which, depending on whose signal is impacted, can cause
nastiness such as:
Transaction timeouts.
Removal of system during GDPS hyperswap.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
9
1
-
9
2

With OA26856, XCF will:
Monitor progress of the path recovery process.
If it takes too long:
Use alternate signalling paths to exchange information with the target
system to determine which “in doubt” signals need to be resent.
Resend the undelivered signals (or a copy thereof) to the target system via
an alternate signalling path.

Enhanced sysplex information display
z/OS 1.9
D XCF,C
format
SYSTEM #@$2 DATA
(base)
85 85 2000 15 10 956

ISOLATE 0 720 50 NO
MAX SUPPORTED SYSTEM-MANAGED PROCESS LEVEL: 16
CF REQUEST TIME ORDERING FUNCTION: INSTALLED

SYSTEM IDENTIFIER: DE502097 050001E7
COUPLEXX PARMLIB MEMBER USED AT IPL: COUPLE00

OPTIONAL FUNCTION STATUS:
FUNCTION NAME STATUS DEFAULT
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
9
3
-
9
4

D XCF,C
SYSTEM #@$3 DATA
With APAR INTERVAL

85
OPNOTIFY
85
MAXMSG
2000
CLEANUP
15
RETRY
10
CLASSLEN
956
OA28603 SSUM ACTION SSUM INTERVAL SSUM LIMIT WEIGHT MEMSTALLTIME

ISOLATE 0 720 70 NO
SIMPLEX SYNC/ASYNC THRESHOLD: 26

DUPLEX SYNC/ASYNC THRESHOLD: 26
SIMPLEX LOCK SYNC/ASYNC THRESHOLD: 26
DUPLEX LOCK SYNC/ASYNC THRESHOLD: 30
Displays XES CF REQUEST TIME ORDERING FUNCTION: INSTALLED
Heuristic algorithm
thresholds


This part of PARTITION: 0A CPCID: 00
response is
SYSTEM IDENTIFIER: DE502097 0A0001DF
unchanged by the
APAR COUPLEXX PARMLIB MEMBER USED AT IPL: COUPLE00


D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
9
5
-
9
6
D XCF,C
SYSTEM SC64 DATA
With z/OS INTERVAL OPNOTIFY MAXMSG CLEANUP RETRY CLASSLEN
R11
165 168 2000 15 10 956

ISOLATE 0 60 10 NO
DEFAULT USER INTERVAL: 165


FDI-related values
SIMPLEX SYNC/ASYNC THRESHOLD: 36

DUPLEX SYNC/ASYNC THRESHOLD: 40
SIMPLEX LOCK SYNC/ASYNC THRESHOLD: 36
DUPLEX LOCK SYNC/ASYNC THRESHOLD: 48
CF REQUEST TIME ORDERING FUNCTION: INSTALLED

With z/OS
R11 SYSTEM STATUS DETECTION PARTITIONING PROTOCOL ELIGIBILITY:
SYSTEM CANNOT TARGET OTHER SYSTEMS.
SYSTEM NODE DESCRIPTOR: 002094.IBM.02.00000002991E

SSD-related SYSTEM IDENTIFIER: 991E2094 060005D9

values NETWORK ADDRESS: N/A
PARTITION IMAGE NAME: N/A
IPL TOKEN: N/A
COUPLEXX PARMLIB MEMBER USED AT IPL: COUPLE00

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
9
7
-
9
8
®
With z/OS
R11
SYSSTATDETECT ENABLED ENABLED
USERINTERVAL DISABLED DISABLED
SSD-related
"FUNCTIONS"


®
Cleaning up "dead" structures

The CFRM Couple Data Sets contain z/OS' understanding of the
contents of every Coupling Facility.
Normally, when a structure is rebuilt from one CF to another,
the CDS is updated as part of the rebuild to accurately reflect
the new location.
If a CF fails (so that it is brought back up with no structures),
the CFRM CDS will maintain infomation about what WAS in the
CF prior to the failure. This information normally remains in the
CDS until the CF reconnects and "CF reconciliation" is performed
The CF tells XCF what structures it actually has.
At this point, the CFRM CDS is updated to reflect the actual contents.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
9
9
-
1
0
0
®

But what happens if you are replacing a CF and don't empty out
the old one before turning it off?
The CF will never reconnect, so the old structure entries can't
get cleaned up.
As long as the CF remains defined in the CFRM policy, a D
XCF,STR command will show those structures in PENDING
DEALLOCATION status.
If you remove the CF from the CFRM policy, the structures will
no longer be displayed, but their entries actually still exist in
the CFRM CDS.


®

To help address this buildup of "dead" structures, z/OS 1.11
provides a new SETXCF command:
SETXCF FORCE,PNDSTR,CFNM=cfname
As long as you issue this command before you remove the old CF
definition from the CFRM CDS, this will clean up all entries
related to that old CF:
Note that at this time, the command will NOT clean out entries related
to CFs that have already been removed from the CFRM CDS.
Only supported on z/OS 1.11, but all levels of z/OS will

understand that CDS has been cleaned up.
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
0
1
-
1
0
2
®
Reminders
THE greatest thing since sliced bread.... SETXCF MAINTMODE
command!
Introduced with z/OS 1.9.
When CF is in maintenance mode:
No new structures will be allocated in CF.
REALLOCATE will move structures that are already allocated in the CF
to an alternate CF per the preference list.
Greatly simplifies planned reconfiguration and maintenance
actions.
Need not update CFRM policy to prevent structures from being
allocated in the CF.


®
Reminders
A pure maintenance procedure can be:
SETXCF START,MAINTMODE,CFNAME=cfname
SETXCF START,REALLOCATE to move structures out of CF
On the CF Console on the HMC, issue the SHUTDOWN command
Perform CF maintenance
Reactivate CF LPAR
SETXCF STOP,MAINTMODE,CFNAME=cfname
SETXCF START,REALLOCATE to restore structures to CF

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
0
3
-
1
0
4
Reminders
SETXCF FUNCTIONS command and COUPLExx keyword -
introduced with z/OS 1.10.
Intent is to let you turn new sysplex functions on or off at the
individual system level:
In 1.10, new FUNCTION was DUPLEXCF16
In 1.11, new FUNCTIONS are USERINTERVAL and SYSSTATDETECT
Future releases will likely deliver more FUNCTIONS
Note that default is NOT always DISABLED - some new
FUNCTIONS will be ENABLED by default (like SSD).
Because COUPLExx member is generally shared by the whole
sysplex, use of FUNCTIONS keyword in COUPLExx should be
used with care (or use system symbol to control at the system
level).
򔻐򗗠򙳰
ibm.com/redbooks Server Time Protocol enhancements
Redbooks Workshop

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
0
5
-
1
0
6
®
STP Status
Server Time Protocol was announced in 2006 and is supported on
all z990 and later generations of processors.
Replaces Sysplex Timer (9037) which has been withdrawn from
marketing (and 9037-001 is withdrawn from service).
Part of the IBM z10 announcement was a statement of direction
that z10 would be the last generation of mainframe to support
connection to Sysplex Timer.
Since its announcement, there have been a series of
improvements to STP function and z/OS support of STP.
IF YOU DON'T HAVE STP TODAY, YOU SHOULD BE

PLANNING FOR IT.

®
STP Enhancements
Prior to z/OS 1.11, some STP-related errors would only be
reported on the HMC.
Prior to z/OS 1.11, no MVS error message was produced if an External
Time Source failed.
To improve the delivery of important information to the operator
and to better integrate with system automation tools, z/OS
1.11 adds new STP-related messages.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
0
7
-
1
0
8
®
STP Enhancements
New IEA031I messages are all ETS-related:
IEA031I STP ALERT RECEIVED. STP ALERT CODE = nn
Alert code is x'01' to x'84'
If you are using an External Time Source with STP, these

messages should be added to your automation.


®
STP Enhancements
The other new message is IEA395I:
IEA395I THE CURRENT TIME SERVER HAS CHANGED TO THE
cccccccccccc (where cccccc is BACKUP or PREFERRED)
This message may not require any action, however ensure that it
generates an alert and that the staff responsible for STP are
aware of the change.
Note that both of these messages will be issued on every

member of the sysplex, so be careful not to have automation
take some action on more than one system.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
0
9
-
1
1
0
®
Changes in support of GDPS K System

The normal course of action when a system loses time
synchronization is to either issue a synchronous WTOR or to
enter a wait state.
However, the most important role of a GDPS K system is to
manage recovery in case of a failure:
If the GDPS K system is stopped it can't drive the recovery processing
The need for time synchronization is to protect the integrity of

the shared data. However, because the K system should never
access any data that is used by the managed systems, it should
be OK for the K system to run in local time mode for a short
time.


®
STP changes in support of GDPS K System

GDPS 3.6 (together with z/OS 1.11) adds support to let a K
system run for up to 80 minutes in local time mode. Requires:
GDPS 3.6
z/OS 1.11 or APARs OA28323 and OA26085 on z/OS 1.9 and 1.10
This support applies to both STP and ETR modes.
The intent is that the system will be available to complete any

Freeze or HyperSwap actions it has initiated and to help you
check the status of sysplex resources.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
1
1
-
1
1
2
®

Normally the system would issue synchronous WTOR IEA015A or
IEA394A when the time signal is lost.
New code in XCF checks to see if this is the GDPS K system:
If not, the normal synchronous WTOR is issued.
If it IS, one of the following messages will be issued:
IEA261I NO ETR PORTS ARE OPERATIONAL. CPC CONTINUES TO RUN
IN LOCAL MODE
IEA381I THE STP FACILITY IS NOT USABLE. SYSTEM CONTINUES IN
LOCAL MODE
And the Master K system keeps running
When time synchronization is restored, the following message is
issued:
IXC438I COORDINATED TIMING INFORMATION HAS BEEN UPDATED
FOR SYSTEM sysname

®

The intent of this support is to improve availability in a
GDPS/PPRC environment, particularly when HyperSwap is being
used:
Note that this support does NOT guarantee that a K system will
survive a timing-related failure. It depends on the sequence in which
things fail.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
1
3
-
1
1
4
򔻐򗗠򙳰
ibm.com/redbooks SMF Logger support enhancements
Redbooks Workshop
SMF Logger support

As the volume of work in a single system continues to increase,
some customers find themselves with an SMF challenge:
They can try to collect all the SMF records they would like (for
auditing, performance, chargeback, capacity planning, security, storage
management, database tuning, and on and on). However sometimes the
system produces SMF records faster than they can be saved to the
SMF SYS1.MANx data set, resulting in some records being discarded.
Or, they can ensure that records are never discarded by not collecting
some record types. This protects the most important records, but
impedes the above activities.
To address this situation, IBM introduced SMF support for
writing to log streams in z/OS 1.9.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
1
5
-
1
1
6
SMF Logger support

On a one-for-one basis, log streams should provide equivalent or
better performance than SYS1.MAN VSAM data sets.
However, you only have one SYS1.MAN data set, but you can
have as many SMF log streams as you like - this removes the
SMF VSAM data set as a bottleneck.
But apart from performance, are there other benefits to using
SMF System Logger support?
Rather than having repositories of ALL SMF record types for ONE
system (SYS1.MANx), you can have repositories of just selected
record types for ALL systems (no more splitting and merging!)

SYSA
SYSA.MANx SYSA.SEQ.SMFALL
ALL
This is how we GIVE

you the SMF data (all
RMF DB2 RACF C/B
SMF types, grouped
GDG GDG GDG GDG
by system)
SYSB This is how you

SYSB.MANx SYSB.SEQ.SMFALL
ALL
USE the data
(grouped by type,
for the sysplex)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
1
7
-
1
1
8
SYSA
What you COULD do

with SMF Logstream RMF DB2 RACF C/B
support LS LS LS LS
SYSB

SMF Logger support

Any other benefits?
Unlike IFASMFDP, you can write 1 record type to multiple repositories
as soon as the records are created.
Don't have to worry about SMF data sets filling up.
Especially where you have partially used, but DUMP REQUIRED data sets
that can't be used after an IPL.
Life cycle management is potentially much easier and more granular if
you switch from GDGs to leaving the data in a log stream.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
1
9
-
1
2
0
®
SMF Logger support

If you decide to exploit this capability, you can:
Just replace the SMF data sets with log streams and maintain all the
downstream processes as they were before.
OR
Have a complete paradigm shift, eliminating the whole GDG process and
keeping the data in log streams instead.


®
SMF Logger support

So what is the catch?
Most installations copy their SMF data from the VSAM SYS1.MANx
data sets to flat sequential data sets as soon as the SMF data set is
switched. These data sets can then be directly accessed by SMF
processing programs such as ERBRMFPP, IFASMFDP, and many others.
Initially, the only way to extract SMF records from the log stream was to:
Write your own extraction program. OR,
Add an intermediate step that would use IFASMFDL to extract the records to a
flat file that the postprocessing programs could then use.
The JCL SUBSYS interface was modified (by APAR OA20995 and
OA29743) to support records larger than 16KB. This means that
programs that previously read the SMF flat file can now read the log
stream directly, using the SUBSYS JCL capability (see the SMF
Manual for more information).

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
2
1
-
1
2
2
®
SMF Logger support

Any others?
Unlike SYS1.MANx data sets, which are typically copied in their
entirety and cleared when they are switched, prior to z/OS 1.11 the
data is kept in the SMF log stream until the RETPD kicks in.
Traditional SMF data sets supported 3 options on the INDD
statement:
DUMP Just copy the data from the input file
CLEAR Delete the data from the input file, but don't copy it
ALL Copy all data from the input file, then delete it all
Prior to z/OS 1.11, SMF log streams only supported the DUMP option - there
was no way (using IFASMFDL) to delete SMF data from the log stream.
As a result, if the same methodology was used to copy data from the
log streams as is used for the SYS1.MANx data sets, the same SMF
records would be extracted multiple times.


®
SMF Logger support

Any others?
In combination with this, IFASMFDL in z/OS 1.9 only supported
absolute dates and times to control what was copied from the log
stream.
The net effect was that you had to change the IFASMFDL control
statements every time it ran, to avoid extracting the same data
multiple times.
z/OS 1.11 (and rolled back to z/OS R9 and R10 with APARs
OA27037 and OA30297) addresses both of these restrictions.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
2
3
-
1
2
4
®
SMF Logger support

Providing the ability to manage the log stream in a way that is
closer to how SYS1.MANx data sets are used:
In IFASMFDL, two new keywords are added:
ARCHIVE Copies the requested data from the log stream and then
deletes it (Similar to the ALL option in IFASMFDP).
DELETE Deletes the identified data without copying it. (Similar
to the old CLEAR option in IFASMFDP).
If you use the ARCHIVE option it is not necessary to provide a specific date
and time range.


®
SMF Logger support

By making it possible to extract data for relative date ranges
without having to change the IFASMFDL parms every time.
IFASMFDL (but not IFASMFDP) now supports relative dates .
As well as supporting the existing DATE keyword, IFASMFDL supports
a new RELATIVEDATE keyword:
Format is RELATIVEDATE(unit,n,y) where:
UNIT is BYDAY, BYWEEK, BYMONTH
n is how many units (days, weeks, or months) to go back (to get the starting point)
y is how many units to go forward from the start date
To get all data for "yesterday", you would specify:
LSNAME(log-stream-name,OPTIONS(ARCHIVE))
RELATIVEDATE(BYDAY,1,1)
Warning - processing is not intuitive, so you should try various
combinations of keywords - also consider MAXDORM values

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
2
5
-
1
2
6
®
SMF Logger support

Anything else?
In order to successfully implement SMF Logger support, you need to
have a reasonably accurate estimate of the volume (in MB) of each
record type that is created each day.
The standard IFASMFDP program is not ideally suited to this role.
To help address this, a set of IFASMFDP exits is available from the

ITSO. In addition to providing more detailed reports, the exits can
also create SMF records with this information which can be
postprocessed.


®
SMF User exits

SMFDPUX1 collects detailed information about data contained into the
INPUT file / Logstreams.
SMFDPUX2 collects detailed information about the data written by

IFASMFDP / IFASMFDL to each OUTPUT file.
SMFDPUX3 uses the information collected by SMFDPUX1 and/or

SMFDPUX2 to optionally produce a set of detailed activity reports
and a set of statistic SMF records
One report and one SMF record for each defined DDNAME /
Logstream

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
2
7
-
1
2
8
Sample JCL

Sample Report about input file(s)
DETAILED ACTIVITY REPORT FOR INPUT DDNAME: MBZIN
RECORD RECORD SYSTEM RECORDS BYTES RECORD RECORD START START END END
TYPE SUBTYP ID READ READ MINLEN MAXLEN DATE TIME DATE TIME
0 N/A SYSA 1 60 60 60 08.362 19:03:43.52 08.362 19:03:43.52

0 1 SYSA 14 40032 2728 2928 08.364 08:15:36.41 08.364 15:05:39.11
2 N/A SYSA 1 18 18 18 08.364 20:05:04.02 08.364 20:05:04.02
3 N/A SYSA 1 18 18 18 08.364 20:05:13.53 08.364 20:05:13.53
4 N/A SYSA 202 53678 215 367 08.362 19:04:40.59 08.364 20:04:31.42
5 N/A SYSA 111 17290 145 157 08.362 19:04:40.01 08.364 20:04:31.46
8 N/A SYSA 1 144 144 144 08.362 19:03:43.01 08.362 19:03:43.01
70 1 SYSA 67 80132 1196 1196 08.362 19:15:00.07 08.364 20:00:00.36
70 2 SYSA 67 10988 164 164 08.362 19:15:01.55 08.364 20:00:00.49
71 1 SYSA 67 109076 1628 1628 08.362 19:15:01.43 08.364 20:00:00.39
72 3 SYSA 1809 1891008 980 1588 08.362 19:15:01.49 08.364 20:00:00.47
72 4 SYSA 73 526476 7212 7212 08.362 19:15:00.37 08.364 20:00:00.30
73 1 SYSA 67 1320168 19704 19704 08.362 19:15:00.25 08.364 20:00:00.37
74 1 SYSA 134 299624 596 3876 08.362 19:15:00.27 08.364 20:00:00.39
74 2 SYSA 73 60444 828 828 08.362 19:15:00.38 08.364 20:00:00.31
74 3 SYSA 73 33288 456 456 08.362 19:15:00.38 08.364 20:00:00.31
74 6 SYSA 73 26572 364 364 08.362 19:15:00.38 08.364 20:00:00.31
75 1 SYSA 268 70752 264 264 08.362 19:15:01.43 08.364 20:00:00.40
76 1 SYSA 536 497416 256 1208 08.362 19:15:01.53 08.364 20:00:00.49
77 1 SYSA 67 10880 160 320 08.362 19:15:01.44 08.364 20:00:00.40
78 2 SYSA 67 126496 1888 1888 08.362 19:15:01.44 08.364 20:00:00.40
80 N/A SYSA 1 244 244 244 08.362 19:04:40.43 08.362 19:04:40.43
81 N/A SYSA 1 2783 2783 2783 08.362 19:03:46.07 08.362 19:03:46.07
89 1 SYSA 196 60004 262 430 08.362 19:15:00.01 08.364 20:00:00.14
89 2 SYSA 196 691572 3526 3594 08.362 19:15:00.01 08.364 20:00:00.16
90 N/A SYSA 8 1966 72 620 08.362 19:03:43.16 08.362 19:04:47.15

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
2
9
-
1
3
0
Sample Report about output file

DETAILED ACTIVITY REPORT FOR OUPUT DDNAME: MBZOUT
RECORD RECORD SYSTEM RECORDS BYTES RECORD RECORD START START END END
TYPE SUBTYP ID READ READ MINLEN MAXLEN DATE TIME DATE TIME
2 N/A SYSA 1 18 18 18 08.364 20:05:30.22 08.364 20:05:30.22

3 N/A SYSA 1 18 18 18 08.364 20:05:35.28 08.364 20:05:35.28
70 1 SYSA 67 80132 1196 1196 08.362 19:15:00.07 08.364 20:00:00.36
70 2 SYSA 67 10988 164 164 08.362 19:15:01.55 08.364 20:00:00.49
71 1 SYSA 67 109076 1628 1628 08.362 19:15:01.43 08.364 20:00:00.39
72 3 SYSA 1809 1891008 980 1588 08.362 19:15:01.49 08.364 20:00:00.47
72 4 SYSA 73 526476 7212 7212 08.362 19:15:00.37 08.364 20:00:00.30
73 1 SYSA 67 1320168 19704 19704 08.362 19:15:00.25 08.364 20:00:00.37
74 1 SYSA 134 299624 596 3876 08.362 19:15:00.27 08.364 20:00:00.39
74 2 SYSA 73 60444 828 828 08.362 19:15:00.38 08.364 20:00:00.31
74 3 SYSA 73 33288 456 456 08.362 19:15:00.38 08.364 20:00:00.31
74 6 SYSA 73 26572 364 364 08.362 19:15:00.38 08.364 20:00:00.31
75 1 SYSA 268 70752 264 264 08.362 19:15:01.43 08.364 20:00:00.40
76 1 SYSA 536 497416 256 1208 08.362 19:15:01.53 08.364 20:00:00.49
77 1 SYSA 67 10880 160 320 08.362 19:15:01.44 08.364 20:00:00.40
78 2 SYSA 67 126496 1888 1888 08.362 19:15:01.44 08.364 20:00:00.40
Helps you decide how much

data (in MB) is created for
each SMF Type and subtype
SMF Logger support

For more information refer to:
z/OS MVS System Management Facility (SMF), SA22-7630
Techdocs presentations (SMF and Logger):
http://www.ibm.com/support/techdocs
CFSIZER utility:
http://www.ibm.com/systems/support/z/cfsizer/
Hot Topics newsletter from August 2007:
http://publibz.boulder.ibm.com/epubs/pdf/e0z2n180.pdf
For the SMF exits, contact Frank Kyne

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
3
1
-
1
3
2
򔻐򗗠򙳰
ibm.com/redbooks Sysplex-related Health Checks
Redbooks Workshop
New Health Checks in z/OS 1.11

DAE_SHAREDSN
Ensure that DAE data set is shared across the sysplex
SVA_AUTOIPL_DEFINED
Checks that hardware supports AutoIPL and, if so, that an AutoIPL
policy exists
SVA_AUTOIPL_DEV_VALIDATION
Checks that the devices referred to in the AUTOIPL policy are "valid"

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
3
3
-
1
3
4
New Health Checks in z/OS 1.11

ZOSMIGREC_STP_TIMER_INUSE
Check that STP is being used
XCF_SYSSTATDET_PARTITIONING
Check that new System Status Detection capability is enabled
XCF_FDI
Check has been changed in support of new calculations regarding spin
loop recovery actions
All health checks are listed and described in chapter 13 of IBM

Health Checker for z/OS V1R11.0 User's Guide.

򔻐򗗠򙳰
ibm.com/redbooks System Logger Enhancements
Redbooks Workshop

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
3
5
-
1
3
6
®
System Logger Enhancements in z/OS 1.11

z/OS 1.11 delivers three enhancements to System Logger:
Improved parallelism for handling connect and disconnect requests for
Staging Data Sets
Improved protection from runaway authorized Logger requestors
System Logger support of the new D GRS,ANALYZE,LATCH command


®
System Logger improved parallelism
System
Logger
A/S
Pre-OS/390 1.3
Allocation Task
Connect / Task1
Disconnect
This is the original System Logger layout - 1

task to handle all Connect and Disconnect
requests

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
3
7
-
1
3
8
®
System
Logger
A/S
OS/390 1.3 to
z/OS 1.10
Allocation Task
Connect / CF Task1
Disconnect CF Task2
CF Task3 In OS/390 1.3, Logger
CF Task4
CF Task5 added 255 more tasks to
CF Task6
.
parallelize connection
. requests for Logger CF
. structures
.
.
CF Task256
DASDONLY Task1

®
System
Logger
A/S
z/OS 1.10
Allocation Task
Connect / CF Task1 DASDONLY Task1

Disconnect CF Task2 DASDONLY Task2
CF Task3 DASDONLY Task3
In z/OS 1.10, Logger . .
. .
added 255 more tasks . .
to parallelize connection . .
requests for .
CF Task256
.
DASDONLY Task256
DASDONLY log streams
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
3
9
-
1
4
0
®

In z/OS 1.11, most of the
System processing that had taken
Logger place in the (single) Allocation z/OS 1.11
A/S
task is moved to the Connect /
Disconnect tasks
Allocation
Task
Connect / CF Task1 DASDONLY Task1
Disconnect CF Task2 DASDONLY Task2
. .
z/OS 1.11 also contains . .
. .
enhancements to more evenly
. .
spread log streams across the
. . DASDONLY Task256
available Connect / Disconnect CF Task256
tasks

®
Measurements
System Logger improved parallelism were taken in a
lab environment
and do not
guarantee that
you would see the
same results in
R11 your environment
R10
R11

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
4
1
-
1
4
2
®

Summary:
The changes in z/OS 1.11 will help any time a connect or disconnect is
done for any log stream that has a staging data set
DASDONLY
Or CF structure that is duplexing to a staging data set
Improvement likely to be most noticeable during recovery of log
streams that have large staging data sets and during system restart
Benefits are automatic - no new parameters to specify, no need for a
new format Logger CDS


®
System Logger protection from runaway tasks

Log blocks can be passed to System Logger in one of two ways:
Synchronously (the requestor doesn't receive control back until the log
block has been saved)
Asynchronously (the requestor passes the log block to Logger and can
proceed as soon as Logger responds)
In z/OS 1.10 (and rolled back with APAR OA14125), System

Logger introduced a cap on the number of log writes that an
unauthorized program can have pending to a given log stream:
The intent was to protect Logger and the system from a runaway task
flooding it with requests and resulting in a shortage of SQA storage

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
4
3
-
1
4
4
®

How is this implemented?
When a requestor passes a log block to System Logger, System
Logger may respond with a Return Code and a Reason Code
(depending on the type of call).
Some Return and Reason codes tell the requestor that the log
stream is temporarily unavailable (for example, the structure
might be in the middle of rebuilding from one CF to another).
All users of Logger should understand these codes and know that
they need to wait and try again "later"
The support in System Logger to throttle the number of pending
requests exploits this existing "temporarily unavailable" concept,
and thereby does not require any additional code in the
requestors.

®

However, many programs that exploit System Logger run
authorized.
To protect the system from runaway authorized requestors,
z/OS 1.11 extends the concept of throttling requests to these
users.
To support the high write rates that are possible on modern
systems, the limit for authorized programs is higher than the
one for unauthorized programs, but the mechanism used to
implement this support is basically the same.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
4
5
-
1
4
6
®

While there is no mechanism for an unauthorized program to be
informed when they can start writing again (they should just
wait a while and try again), authorized programs can either use
the same mechanism, or they can listen for an ENF 48 signal
from Logger to inform them that they can start writing again.


®

To alert you to this activity, Logger writes LOGREC records
whenever the theshold for a log stream is exceeded, and again
when it is relieved:
5752SCLOG RIDS/IXGF2WRT RIDS/IXGINLPA#L LVLS/730 FLDS/RETCODE
VALU/H00000004 FLDS/REASON VALU/H04030017 IXGF2WRT ASYNC LIMIT
REACHED
5752SCLOG RIDS/IXGF2WRT RIDS/IXGINLPA#L LVLS/730 FLDS/RETCODE
VALU/H00000004 FLDS/REASON VALU/H04030018 IXGWRITE ASYNC LIMIT
RELIEVED

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
4
7
-
1
4
8
®

Summary:
This change should improve avalability by protecting the system from
runaway Logger exploiters, or in customers that are very heavy users
of Logger and that experience occasional delays within Logger
processing
No change or customization is required to enable this new function
Owners of code that writes to a log stream may optionally enhance their
code to understand the new reason codes from System Logger


®
System Logger serviceability enhancement

Like most products, System Logger uses a combination of ENQs
and latches.
If you experience a delay in System Logger prior to z/OS 1.11,
you could use the D GRS,ANALYZE command to determine if
there were any ENQ delays that might be impacting Logger
However, trying to obtain information about latch contention was
difficult.
Frequently, the identification of the latch problem would only be done
by looking in a dump - too late to help resolve the problem in real time

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
4
9
-
1
5
0
®

In z/OS 1.11, GRS has been enhanced to provide similar
information and analysis for latches that it previously provided
for ENQs.
New D GRS,ANALYZE,LATCH,* may help you identify the log
stream that is involved in any contention:
Note that the holder of a latch (System Logger in this case) must
provide information about which resource a given latch is associated
with in order for GRS to be able to display that information


®

SY1 d grs,an,latch,depend,detail
SY1 ISG374I 14.23.53 GRS ANALYSIS
DEPENDENCY ANALYSIS: ENTIRE SYSTEM
----- LONG WAITER #1
JOBNAME: IXGLOGR (ASID=002B, TCB=005DDE88)
REQUEST: EXCLUSIVE LT:7F42907800000001 WAITING 00:01:33 FOR RESOURCE (CREATOR ASID=002B)
SYS.IXGLOGER_LCBIT___CTA:00000003_SLSA:0001 LST:7F42BD00000000BD2:SOME.LOG.STREAM
JOBNAME: WRITE3 (ASID=0026, TCB=005D3A08)
REQUEST: SHARED LT:7F42901000000002
ANALYSIS ENDED: THIS UNIT OF WORK IS NOT WAITING
In this example system logger (jobname IXGLOGR) is waiting to

get an exclusive latch for log stream SOME.LOG.STREAM, but
another job (WRITE3) holds the latch shared
This information may help you resolve the delay. At a minimum,

it will expedite problem determination by identifying which
resource is in contention
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
5
1
-
1
5
2

The log stream name can then be used to get more information
from Logger
SY1 d logger,c,lsn=SOME.LOG.STREAM,detail
SY1 IXG601I 15.46.06 LOGGER DISPLAY 912
CONNECTION INFORMATION BY LOGSTREAM FOR SYSTEM SY1
LOGSTREAM STRUCTURE #CONN STATUS
--------- --------- ------ ------
SOME.LOG.STREAM LIST06 000002 IN USE
DUPLEXING: STRUCTURE
GROUP: PRODUCTION
JOBNAME: WRITE3 ASID: 0026
R/W CONN: 000000 / 000001
RES MGR./CONNECTED: *NONE* / NO
IMPORT CONNECT: NO
JOBNAME: NJCONC ASID: 001B
R/W CONN: 000001 / 000000
RES MGR./CONNECTED: *NONE* / NO
IMPORT CONNECT: NO
NUMBER OF LOGSTREAMS: 000001
A detailed Logger display of the log stream connections shows that there
are two connectors on this log stream

򔻐򗗠򙳰
ibm.com/redbooks
System-Managed Duplexing performance
experiences
Redbooks Workshop

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
5
3
-
1
5
4
®
System-Managed Duplexing
As more customers move towards an ICF-only configuration,
interest in System-Managed Duplexing is increasing. In order to
give customers some idea of the impact of enabling
System-Managed Duplexing, the ITSO ran some measurements
to compare a CICS/DB2 workload with a duplex and simplex lock
structure.


®
Why we need System-Managed Duplexing

CF1 CF2
Standalone DB2 Lock

CF structure
DB2A DB2B
ZOSA ZOSB

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
5
5
-
1
5
6
®

CF1 CF2
DB2 Lock
structure
Standalone
CF
DB2A DB2B
(no abend, (no abend,

DB2A DB2B
keeps keeps
running) running)
ZOSA ZOSB


®

CF1 CF2
Standalone
CF
DB2A DB2B
(no abend,
DB2B
keeps
running)
ZOSA ZOSB

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
5
7
-
1
5
8
®

CF1 CF2
ICF
DB2A DB2B
abend
ZOSA ZOSB


®

CF1 CF2
ICF
DB2A DB2B
(no abend,
DB2B
keeps
running)
ZOSA ZOSB

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
5
9
-
1
6
0
®
Our configuration....
– z10 model E26

– 15 CPs
FACIL03
MVS 0A – 2 IFLs
– 6 ICFs
– 256 GB Memory
– 8 x ISC-3
– 2 z/OS LPARs with 2

dedicated LCPs each
– 2 CF LPARs with 1
dedicated ICF each
MVS 02
– DS8000 DASD
• No mirroring
FACIL04
2x IC 2x ISC


®
DB2/CICS workload was used to generate a load on DB2 and the
CF
Tested with simplex DB2 lock and SCA structures, then again
with both structures duplexed
z/OS configuration was:

z/OS LPARs had 2 dedicated CPs
Only ISC links were used to connect to the “other” CF, ICP links used
to connect to local CF
CFs were connected to each other using ISC links
Mix of light, medium, and DB2-intensive transactions
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
6
1
-
1
6
2
Impact of SM Duplexing across all transactions
200 18
160 152.6 152.3

13.5
12
120
9
95.5 9
84.4
80
4.5
40
22
6.4
0 0
Simplex Duplex
z/OS CPU % CF CPU % Trx Resp Time (msecs) Trx/sec

( Relative to 1 CP )

Transaction rate nearly unchanged – 152.3 txns/sec with
duplexed lock structure vs 152.6 for simplex
Overall transaction response time increased by 3 ms, from 9ms

to 12ms
z/OS CPU utilization increased from 84% to 95%
Combined CF CPU utilization (across both CFs) increased from

6.4% to 22%

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
6
3
-
1
6
4
Let’s look a little more closely at a DB2-intensive transaction,
and at the activity to the CF
The most DB2-intensive transaction (and therefore the one most

likely to be impacted by the use of System-Managed Duplexing)
was TPNO, so we will look at that more closely
We should also look at the activity of the DB2 lock structure in

the CF

Impact of SM Duplexing on TPNO transaction
150 6200
6067
6039
120 6000
106
90 5800
60 5600
41
26.5
30 5400
29
26.6 26.6
0 5200
Simplex Duplex
Trx/sec Trx Resp Time (msecs) Lock Req/sec Lock Serv Time (usecs)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
6
5
-
1
6
6
®
The response time of the DB2 lock structure increased from an
average of 26 us. up to 110 us.
However, the transaction rate for the TPNO transaction was
identical – 26.6 per second with duplexed or simplex lock
structure
And the TPNO transaction response time increase was less than
might be expected considering the change in the CF response
times – they went up from 29ms to 41 ms
DB2 lock structure request rate was nearly unchanged – 6067
(simplex) vs. 6039 (duplexed)


®
Even though there was a significant increase in lock request
response time, the change in the overall transaction time was
reasonable. Even for the DB2-intensive transaction, the
increase was reasonable.
However, while the acquisition cost of ICFs is less than an
external CF, don’t forget that the use of System-Managed
Duplexing will increase CF CPU utilization, z/OS CPU utilization
(which you pay software licenses on), and CF subchannel
utilization.
Also, the availability delivered by an ICF, even with
System-Managed Duplexing, in case of a CPC failure is NOT the
same as that achievable with an external CF.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
6
7
-
1
6
8
򔻐򗗠򙳰
ibm.com/redbooks
Mean Time to Recovery
Best Practices
Redbooks Workshop
Background
In response to growing business pressure to get applications back
up and running as quickly as possible after an IPL, especially as
systems get larger and larger, IBM have initiated a multi-year
initiative to improve System z Mean Time To Recovery (MTTR).
In support of that initiative, the ITSO ran a residency in May
2009 to create a Redbook documenting MTTR Best Practices.
Residency was staffed by representatives from the CICS (UK),
DB2 (China), IMS (Texas), WAS (Raleigh), and z/OS labs, and
renowned consultant Cheryl Watson (the queen of SMF!).
Also, in tandem with the residency, we studied IPL data from
about 150 customer and IBM systems
This is referred to throughout this session

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
6
9
-
1
7
0
®
Residency scope
The objectives of the residency were:
Investigate ways for customers to better exploit the existing
technology to improve their MTTR
Identify MTTR-related changes in recent releases of z/OS or the
major subsystems
Work with Development organizations to identify potential areas for
improvement for future releases
Create a Redbook to help customers benefit from our experiences
Note that the objective of this session (and the Redbook) is not
to demo product enhancements - it is to help you reduce your
MTTR
We tried NOT to make any judgements about how much of a
saving is worthwhile
ATM Card
1234 5678 9012

VALID FROM GOOD THRU
XX/XX/XX
XX/XX/XX XX/XX/XX
XX/XX/XX
PAUL FISCHER
PAUL FISCHER


®
Our software configuration

z/OS R9 and R10 running on a z10
CICS TS 3.2 and CICSPlex/Systems Manager
DB2 V9
IMS V10
WebSphere V7

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
7
1
-
1
7
2
Our configuration
2097-E26
FACIL03 FACIL04
z/OS R9 #@$2 z/OS R10 #@$3 z/OS R10 #@$A
CICS DSTCPc2* CICS DSTCPc3* CICS DSTCPcA*

DB2 D8Q1* DB2 D8Q2* DB2 D8QA*
IMS IM1A* IMS IM2A* IMS IM3A*
WAS7 T2*
Storage 8GB Storage 8GB Storage 8GB

CPs 2 Dedicated CPs 2 Dedicated CPs 2 Dedicated

Exactly what is the scope of MTTR?

business impact
all business processes available applications

problem occurs
middleware
possibly orderly shutdown
shutdown subsystems restart
z/OS
TIME Æ
For Unplanned - gather diagnostic information
For Planned - implement planned changes

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
7
3
-
1
7
4
®
Systems Management
Yes, we know....... BORING......
However, cleaning up WHAT you do may deliver more significant

benefits than tuning how you do it....
Plus, in most cases, there are no obvious low-hanging fruit. And
most of the low-hanging fruit that we investigated were related
to "unusual" practices.
The "fix" for these situations was a change in how or what they were
doing rather than a technology fix


®
Systems Management
But before you try to fix anything, does everyone in your
installation agree on exactly what is included in the "down time"
value? Do the MVS, and DB2, and Network, and operations
staff all have the same view of the scope of the outage
window?? A lot of time can be wasted by people talking about
different things but thinking they are talking about the same
thing.
MAYBE you don't actually have a problem at the moment. But
this is a good opportunity to start tracking your restart times,
so you can be proactive if they start increasing.
We provide some simple programs on an as-is basis that might help you
with this

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
7
5
-
1
7
6
®
Systems Management
Some examples of where changes in systems management
practices could help MTTR....
WTORs during IPL. In the IPLs we looked at across many
configurations, the systems that had the largest elapsed times were
nearly all due to waiting for an operator to respond to a WTOR.
Time-driven startup and shutdown... All startup and shutdown
commands should be initiated because a particular message was issued,
NOT because x minutes or seconds had elapsed since some prior event.
Waiting for EVERYTHING (every last little STC) to be stopped in an
orderly manner.
NOT waiting for some STCs (like database managers) to stop in an
orderly manner.
ROT is that recovery from an incomplete shutdown takes (at least) twice as
long as a clean shutdown would have taken

®
Systems Management (cont)

And some more....
Which takes longer? To take your shoe off and then decide that you
didn't need to take it off and put it back on again? Or not to take it
off in the first place? How about IPLing a system to change
something that could have been changed dynamically? Or just making
the change dynamically and avoiding the IPL?
As a starting point, review Redbook "z/OS Planned Outage Avoidance
Checklist", SG24-7328
z/OS 1.11 adds a new SETALLOC command, removing another reason for
doing IPLs

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
7
7
-
1
7
8
®

Some more examples....
How much thought went into the sequence in which you start and stop
address spaces? Does your batch scheduling product REALLY need to
be initialized before CICS?
Objective should be to make the online applications available as quickly as
possible. The initialization of anything that the applications do not need
should be postponed until after the critical apps are up and running.
Rolling IPLs are the way to go. However, if you MUST IPL multiple
systems together, give some thought to which ones are IPLed in parallel
Try to avoid CPU contention, so concurrently IPL systems on different CPCs
if possible.
Doing DASD I/O accounts for most of the time to IPL, so concurrently IPL
systems that share the same sysres (so subsequent systems get the benefit
of the first one pre-loading the required tracks into disk subsystem cache).


®

Better exploiting the available technology. For example....
Some customers still do IPLs by logging onto the HMC and initiating the
LOAD from there. This is a time-consuming process.
As an alternative (if your automation doesn't provide the capability) what
about using the AutoIPL capability in z/OS R10 to eliminate the
Operator-HMC interaction during the IPL process?
V XCF,sysname,OFFLINE,REIPL
Elapsed time from V XCF,xxx,OFFLINE,REIPL to IEA371I was one minute 18
seconds for a system with CLEANUP time of 15 seconds

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
7
9
-
1
8
0
®

You have to know your starting point.
Before you can address MTTR, you need to know where the time is being
spent today
If you don't know this, how can you know what to tune?
If you don't know your starting point, and how consistent it is, how can you know
whether a change you make actually helps or hurts?
While Systems Management may not be exhilarating, what is the

point in tuning a process that isn't needed in the first place?


®
Systems Management (sort of)

1) "I have to stop my batch initiators 2 hours before I IPL."
2) "My planned outage window includes doing a database reorg."
3) "We IPL so infrequently (because the users won't let us IPL
more often) that we hit operational problems every time."
MTTR will never be able to make these problems go away. But
Data sharing could help item 1.
Fully exploiting facilities like online reorg might make item 2 less
painful
Data sharing should give you the OPTION to IPL more frequently
(because an IPL should not mean an application outage)
Exploiting IPL avoidance techniques might help
But all of these are controlled by YOU, not IBM.....
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
8
1
-
1
8
2
®
OK, but what about the good stuff?

We looked at the startup times of many systems in
addition to ours and had some observations:
There are parts of the IPL process that consistently take
"longer" across all systems
However for those parts, the range of times can vary widely
from one system to another
There is no such thing as a "typical system"
Therefore, in order to know which topics are likely to be
fruitful for you, you first need to understand which are the
"big hitters" in your environment


®
Where are you spending all your time?
Based on observations across about 150 systems

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
8
3
-
1
8
4
Big hitters
Previous chart is based on IPLSTATS reports for about 150
systems
IPLSTATS program formats the IPST control block which contains
information about the elapsed time for most programs executed prior
to the end of Master Scheduler Initialization
You can also use the IPCS VERBX BLSAIPST MAIN command to format the
control block
Chart shows the modules with a MEDIAN time of > 1 sec
Nearly all these modules, especially those with very long

maximum times, involved either many I/Os, or operator
interaction (replying to a WTOR)

IPL Sequence Typical elapsed times
We will focus
Hardware IPL
on these parts IPL RIMs (3-45 seconds)
of the IPL NIP RIMs (22-525 seconds)
MSI Part 1 (0.5-6 seconds)
MSI Part 2 (2-368 seconds)
IEACMD00 COMMND00
But not about
And talk a Automation these
little about JES2
these OMVS
SMS CICS
VTAM/TCP DB2
IMS
®2009 IBM Corporation. All rights reserved. WAS
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
8
5
-
1
8
6
®
Phase 1 - Hardware IPL

This phase starts when the LOAD function is selected on the
HMC and completes with the following message:
The elapsed time was largely unaffected by configuration

changes such as the number of CPs, or the amount of storage in
the LPAR, or the use of LOAD NORMAL vs LOAD CLEAR
However it does fluctuate a little normally from one IPL to another

®
Phase 2 - IPL RIMs

IPLST000I z/OS 01.10.00 #@$A 20970019DE50
IPLST001I IPL started at: 2009/05/05 21:27:58.972
2 CPs IPLSTATS
IPLST100I **** IPL Statistics ****
IEAIPL10
IEAIPL20
0.001
0.000
ISNIRIM - Read SCPINFO
Test Block storage to 2G
report
IEAIPL11 0.029 Fast FIND service
IEAIPL31 0.005 LOAD service
IEAIPL30 0.001 IPLWTO service
The sequence of events in the

IEAIPL46 0.124 Read SCHIBs into IPL workspace
IEAIPL49 0.000 Process Load and Default parameters
report reflects the sequence

IEAIPL50 0.009 IPL parmlib - process LOADxx and NUCLST
Displays the first message on the console
IEA371I SYS0.IPLPARM ON DEVICE D056 SELECTED FOR IPL PARAMETERS
IEA246I LOAD ID FK SELECTED
IEA246I NUCLST ID $$ SELECTED
in which the processes are
IEAIPL51
IEAIPL43
0.000
0.003
System architecture
Find and Open IODF data set
started*
IEA519I IODF DSN - IODF.IODF02
IEAIPL60 0.000 Read NCRs from IODF
IEAIPL70 0.164 UIM environment - load CBD and IOS services
IEAIPL71 0.033 Build DFT for each device Longer running processes
IEAIPL08 0.001 Read EDT information from IODF
IEAIPL40 0.050 Read MLTs from nucleus
IEAIPL42 0.018 Read NMLs from nucleus (IEANynnn modules Related console messages and
IEAIPL41 0.645 Read PDS directory entries and CESD records notes (These are not part of
Time is dependent on number of parmlibs specified in LOADxx
IEAIPL05 0.000 Build and sort NUCMAP the IPLSTATS report)
IEAIPL02 1.728 Load nucleus modules
IEA091I NUCLEUS 1 SELECTED
IEAIPL04 0.021 Allocate PFT and SQA/ESQA
IEAIPL14 0.000 Build LSQA/ELSQA for Master
IEAIPL09
IEAIPL07
0.042
0.006
IAXMI - PFT, master RAB, etc.
Update AMODE for nucleus resident SVCs *This report is from an IBM test
IEAIPL03
IEAIPL18
0.015
0.015
Build UCBs, ULUT, etc.
Copy and relocate EDT to ESQA system, so the times should not be
IEAIPL99 0.211
3.121
Page frame table and cleanup
TOTAL IPL TIME (seconds)
considered representative
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
8
7
-
1
8
8
ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰®
Phase 2 - IPL RIMs

These are all
z/OS 1.10 I/O related
IPL RIM Largest Com ponents
makes this
asynchronous
20.0
18.0
16.0
14.0
12.0
Min
10.0 Median
Max
8.0
6.0
4.0
z/OS 1.10 2.0
optimizes this
0.0
Test Block st orage t o Build and sort IPL parmlib - process Read PDS direct ory Load nucleus modules
2G NUCMAP LOADxx and ent ries and CESD
NUCLSTxx records
Ev e n t
ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰®
Phase 2 - IPL RIMs

Hints and tips:
Make sure there are no RESERVEs on sysres or IODF volumes
Most of the elapsed time is I/O-related, so using the fastest DASD
possible will help all these steps
Recommend using ** for the IODF suffix in the LOADxx member
But make sure that the new IODF data set is available to every LPAR before
doing the hardware ACTIVATE
z/OS 1.10 moves the storage validation process out of the critical path
This can save up to 15 seconds off the IPL, depending on the configuration
z/OS 1.10 also optimized the Build NUCMAP process - was up to 18
seconds, but was .001 or less on all 1.10 systems we studied
This is part of a larger effort to reduce the number of times
SYS1.NUCLEUS is processed

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
8
9
-
1
9
0
®
Phase 3 - NIP RIMs IEAVNP11 0.044 Locate and Open master catalog
IEA370I MASTER CATALOG SELECTED IS MCAT.V#@$#M1
IEAVNPC7 0.012 Open SYS1.SVCLIB
NIP started at: 2009/05/14 22:43:31.730
IEAVNPOP 0.008 Open PARMLIB
**** NIP Statistics **** IEE252I MEMBER IEASYMXX FOUND IN SYS1.PARMLIB
IEAVNIP0 0.011 NIP Base
IEA008I SYSTEM PARMS FOLLOW FOR z/OS 01.09.00 HBB7740 019
IEAVNIPM 0.099 Invoke NIP RIMs
IEASYS00
IEAVNPE6 0.108 Service Processor Interface
IEE252I MEMBER IEASYS00 FOUND IN SYS1.PARMLIB
IEAVNPFF 0.028 Loadwait/Restart
IEAVNPA6 0.009 RTM - RTCT and recording buffer IEA007I STATIS SYSTEM SYMBOL VALUES 025
IEAVNPC6 0.014 WTO &SYSALVL. = “2”
IEAVNPC3 0.015 Issue messages from IPL message queue This is followed by all of the symbols from IEASYSxx
IEAVNP24 0.029 SMS Open/Mount IEAVNPIL 0.041 Process IEALSTxx
IEAVNP06 0.017 Machine Check IEAVNPC4 0.096 Prompt for System Parameters
IEAVNP27 0.028 Reconfiguration IEAVNP03 0.017 Merge and analyze system parameters
IEAVNPA2 25.697 IOS - Non-DASD UCBs IEAVNPCF 4.491 Process system name and system variables
IEAVNPCA 0.301 NIP Console The time for this step is dependent on the size of IEASYMxx
IEAVNPB2 3.260 IOS - DASD UCBs IEAVNP76 0.032 Open LOGREC
IFB086I LOGREC DATA SET IS SYS1.#@$2.LOGREC 060
IEAVNPE8 0.051 RSM - Process REAL=
IEAVNP23 0.012 Build GRS blocks in SQA
IEE252O MEMBER GRSCNF00 FOUND IN SYS1.PARMLIB
ISG313I SYSTEM IS JOINING A GRS STAR COMPLEX. RING CONFIGURATION
KEYWORDS IN GRSCNF00 ARE IGNORED.
IEE252I MEMBER GRSRNL00 FOUND IN SYS1.PARMLIB
IEAVNP04 0.121 ASM - Open page and swap data sets
IEA940I THE FOLLOWING PAGE DATA SETS ARE IN USE:
PLPA ........... - PAGE.#@$2.PLPA
COMMON ......... - PAGE.#@$2.COMMON
LOCAL .......... - PAGE.#@$2.LOCAL1
IEAVNPA8 0.009 VSM - Expand SQA
IEAVNP14 0.121 ASM part 2 - Build SQA control blocks
IEAVNPGD 0.002 Move console data to ESQA
IEAVNP25 0.007 Process SVC=
IEE252I MEMBER IEASVC00 FOUND IN SYS1.PARMLIB
SVCPARM 213,REPLACE,TYPE(2) /* IMS TYPE 2 SVC */
IEAVNP05 9.828 LPA, APF
The length of this step depends on whether CLPA was specified


®
Phase 3 - NIP RIMs

IEAVNP44 0.008 ASA Reuse stuff IEAVNPF9 21.169 XCF
IEAVNPB1 0.002 Process CSCBLOC= The XCFAS address space is created here.
IEAVNPE2 0.003 RACF SAF
IEE252I MEMBER COUPLE12 FOUND IN SYS1.PARMLIB
IEAVNPB8 0.011 Create CSA
IEE252I MEMBER CRIXCF00 FOUND IN SYS1.PARMLIB
IEAVNP47 0.002 ENF
IEAVNPD6 0.002 RTM - SDUMP, ABDUMP, ESTAE IEE252I MEMBER CRIXES00 FOUND IN SYS1.PARMLIB
IEE252I MEMBER DIAG00 FOUND IN SYS1.PARMLIB IXL157I PATH A0 IS NOW OPERATIONAL TO CUID: FFE1 117
IEAVNP09 0.002 Build ASVT COUPLING FACILITY 002097.IBM.02.00000001DE50
IAR013I 4,096M STORAGE IS RECONFIGURABLE PARTITION: 1E CPCID: 00
IEAVNPD8 5.864 RSM - Frame queues, VRREGN= and RSU= . all paths are shown
This time is dependent on the size of storage and not RSU IXC306I START PATHOUT REQUEST FOR STRUCTURE IXC_DEFAULT_1 124
IEAVNP10 0.030 SRM - OPT=, IPS=, etc. COMPLETED SUCCESSFULLY: PARMLIB SPECIFICATION
IEE252I MEMBER IEAOPT00 FOUND IN SYS1.PARMLIB . all PATHINs and PATHOUTs for all structures are processed here
IEAVNPD1 0.007 ABDUMP IXC466I INBOUND SIGNAL CONNECTIVITY ESTABLISHED WITH SYSTEM #@$A
IEE252I MEMBER IEAABD00 FOUND IN SYS1.IBM.PARMLIB VIA STRUCTURE IXC_BIG_2 LIST 8
IEE252I MEMBER IEADMP00 FOUND IN SYS1.PARMLIB . all signals for all systems and structures processed here
IEE252I MEMBER IEADMR00 FOUND IN SYS1.IBM.PARMLIB IXC286I COUPLE DATA SET 184
IEAVNPD2 0.020 SDUMP SYS1.XCF.MAXSYS12.CFRM01,
IEAVNPCX 0.002 Context services, registration VOLSER #@$X2, HAS BEEN ADDED AS THE PRIMARY FOR CFRM ON SYSTEM #@$2
IEAVNPX1 0.002 NIP cleanup . all couple data sets processed here, including SFM if used
IEAVNPF5 0.043 PCAUTH IEAVNP33 1.442 GRS
This is ASID=0002, the 2nd address space created after *MASTER* IEE252I MEMBER CTIGRS00 FOUND IN SYS1.IBM.PARMLIB
IEAVNPF8 0.033 RASP IXL014I IXLCONN REQUEST FOR STRUCTURE ISGLOCK 206
This is ASID=0003 WAS SUCCESSFUL. JOBNAME: GRS ASID: 007
IEAVNP1F 0.041 SRM - I/O measurement blocks CONNECTOR NAME: ISGLOCK##@$2 CFNAME: FACIL03
IEAVNPC2 0.011 IOS - Move CDT to SQA ISG337I GRS LOCK STRUCTURE (ISGLOCK) CONTAINS 4194304 LOCKS.
IEAVNP51 0.023 TRACE IEAVNPED 0.021 PROD
This is ASID=0004 IEE252I MEMBER IFAPRD00 FOUND IN SYS1.PARMLIB
IEAVNP20 0.013 Process CLOCK= ISG300I GRS=STAR INITIALIZATION COMPLETE FOR #@$2.
IEE252I MEMBER CLOCK00 FOUND IN SYS.PARMLIB IEAVNP26 3.659 SMS
IEAVNP21 0.057 TOD clock IEE252I MEMBER IGDSMS00 FOUND IN SYS1.PARMLIB
IEAVNP57 0.008 SDUMP IEE252I MEMBER CTISMS00 FOUND IN SYS1.IBM.PARMLIB
This creates the DUMPSRV address space, ASID=005
PDSE processing produces several messages at this point
Address spaces SMSPDSE and SMSPDSE1 are created
Optionally, address space SMSVSAM is created

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
9
1
-
1
9
2
®
Phase 3 - NIP RIMs

IEAVNPE5 3.590 LNKLST
IEAVNPD5 0.821 Load pageable device support modules
IEAVNP88 0.112 Allocation move EDT II
IEAVNPA1 1.829 CONSOLE
The CONSOLE address is created here
IEE252I MEMBER CONSOL00 FOUND IN SYS1.PARMLIB
IEA549I SYSTEM CONSOLE FUNCTIONS AVAILABLE 240
IEAVNPDC 0.533 WLM
The WLM address space is created here, and WLM is initialized
IEAVNP16 2.722 EXCP appendages
IEE252I MEMBER IEAAPP00 FOUND IN SYS1.PARMLIB
IEAVNP13 0.033 Prepare NIP/MSI interface
IEAVNP17 0.002 GTF Monitor Call interface
IEAVNPG8 0.004 VSM defined monitor call enablement
IEAVNP18 0.099 PARMLIB Scan Routine interface
IEE252I MEMBER IEACMD00 FOUND IN SYS1.PARMLIB
IEE252I MEMBER COMMND$2 FOUND IN SYS1.PARMLIB
IEAVNPF2 0.083 Process IOS=
IEAVNP15 0.350 Process VATLST
IEAVNPRR 0.002 RRS
IEAVNPOE 0.415 USS
IEAVNPSC 0.010 Metal C RTL
IEAVNPLE 0.060 System LE RIM
Language services are initialized here
UNIX system services are initialized here
OMVS address space is created here
IEAVNPUN 0.071 Unicode
IEAVNPXL 0.013 zXML Parser
IEAVNP1B 0.117 Close catalog
CATALOG address space is created here
IEAVNIPX 0.001 NIP final cleanup
94.128 TOTAL NIP TIME (seconds)


®
Phase 3 - NIP RIMs

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
9
3
-
1
9
4
®
Phase 3 - NIP RIMs

Hints and tips
This is typically the largest part of the IPL process
Of the 10 modules in this phase with median elapsed times of over 1
second, 7 of them are I/O intensive
Try to avoid having devices online that you do not need. Especially,
don't have devices come online, and then have automation vary them
offline!
Avoid RESERVEs!!
An amount of the processing involves looking at every online DASD
volume, so the fewer volumes you have, the less time this should take
(use of Mod 54s instead of Mod 9s).
Consider whether to do a CLPA with every IPL.
Maximum value for LPA step was >200 seconds
Changing our system to NOT do CLPA reduced this step to <1 second.

®
Phase 3 - NIP RIMs

At IPL time, XCF reads the entirety of every CDS to validate the
contents. So, the larger the CDS is, the longer it takes to process
it.
Impact of Sysplex CDS size
100 Downside is
90
that making a
CDS smaller
80
70
60 requires a
50 Seconds sysplex IPL
40
30
20
10
0
Average Run 1 Average Run 2 Average Run 3 Maximum

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
9
5
-
1
9
6
®
Phase 3 - NIP RIMs

Hints and tips
For systems with ETRMODE(YES), initializing TOD clock can take 10
seconds or longer. Use of STPMODE may reduce this.
Avoid duplicate volsers, which require time-consuming operator
responses (not to mention confusion).
Do not prompt the operator for system parameters (IMSI parameter
on the IPL load parm).
Only define consoles that you actually need and use. If all systems in
the sysplex are running z/OS 1.10, consider use of distributed mode
consoles.
Try to avoid having very large Parmlib concatenation, or large
concatenations of Parmlib members.
Remove "dead" systems from LOADxx and IEASYMxx members


®
Phase 3 - NIP RIMs

Hints and tips
Some enhancements in IOS in z/OS R11
IOS function R10 R11

(min:sec) (min:sec)
Non-DASD UCBs 2:25.500 0:31.744
DASD UCBs 1:09.758 0:25.213
Dynamic pathing 3:08.414 1:39.979

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
9
7
-
1
9
8
®
Phase 4 - MSI Part 1

**** IEEVIPL Statistics ****
IEETRACE 0.003 Master trace
ISNMSI 0.758 SPI
UCMPECBM 0.996 CONSOLE address space
ENFPC005 0.000 CONSOLE ready ENF
IEFSCHIN 0.256 IEFSCHAS address space
IEFJSINT 0.003 Subsystem interface
IEFSJLOD 0.015 JESCT
IAZINIT 0.059 JESXCF address space
IAZFSII 0.012 FSI trace
IEFQBINT 0.021 SWA manager
IEFAB4I0 0.194 ALLOCAS address space
IEEVIPL 2.334 Uncaptured time: 0.000
4.667 Total for MSI phase 1
Hints and tips

Very little to say - maximum time in this phase was about 6 seconds...


®

MSI started at: 2009/05/14 22:45:08.870
Bring additional CPs

**** IEEMB860 Statistics ****
ILRTMRLG 3.079 ASM
IECVIOSI 17.735 IOS dynamic pathing
online around here

IEE252I MEMBER IEAAPP00 FOUND IN SYS1.PARMLIB
IEC336I STORAGE SUBSYSTEM X’89E3’ INITIALIZED
IEC336I STORAGE SUBSYSTEM X’89E5’ INITIALIZED
. . . plus one message for each subsystem
ATBINSYS 0.025 APPC
IKJEFXSR 0.161 TSO
IXGBLF00 0.015 Logger
*IFB081I LOGREC DATA SET IS FULL,16.06.21, 360
DSN=SYS1.#@$A.LOGREC
COMMNDXX 31.315 COMMANDxx processing
Most of this time is executing the immediate commands
The rest of the CPUs are varied online at this point
SMFWAIT 116.122 SMF
IEE252I MEMBER SMFPRM00 FOUND IN SYS1.PARMLIB
Initialize SMF, create SMF address space
SMF will create all of the SMF IPL records at this time
SECPROD 3.473 Security server
IEE712I CONTROL PROCESSING COMPLETE
ICH508I ACTIVE RACF EXITS: NONE
IEFJSIN2 6.733 SSN= subsystem
IEFHB4I2 0.013 ALLOCAS - UCB scan
CSRINIT 0.011 Windowing services
FINSHMSI 0.077 Wait for attached CMDs
IEEMB860 10.630 Uncaptured time: 0.100
178.759 Total time for this phase
277.554 TOTAL IPL TIME (seconds)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
1
9
9
-
2
0
0
MSI Phase2 Largest Component s
250
200
150
MIN
MEDIAN
MAX
100
50
0
SMF SSN=subsyst em Securit y server IOS dynamic pat hing
Event


Hints and Tips:
The last part of the IPL process, but potentially a significant portion
of the IPL (up to 368 seconds)
SMF processing median time is a small fraction of the maximum. Two
main contributors to the difference were:
Specifying PROMPT in SMFPRMxx
Collecting SMF Type 19 records
Does anyone use these? There are other, less intrusive, ways to get the same data
Elapsed time for the IEFSSN processing depends on how many
subsystems are being defined
Subsystems are defined serially, in the order in which they are defined
Reported time includes the time for each subsystem to initialize
Recommend that you use the new format of IEFSSN (keyword) to ensure
you benefit from any possible future enhancements
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
0
1
-
2
0
2
®

Hints and Tips:
See APAR OA27436 for info about how to set up IECIOSxx to reduce
the time required to set up the MIH tables
Advertised to help Vary On processing, but can also help IPL
IOS Dynamic Pathing:
z/OS R11 changed the Dynamic Pathing algorithm to make it more efficient
The factors that influence the elapsed time for this processing are:
Number of DASD devices
Number of paths to each device
Number of PAV base devices
Number of PAV aliases
If using IRD Dynamic CHPID Management, paths only get validated when
they are assigned, so implementing DCM (and changing some paths from
static to dynamic) MAY save you some time during IPL


®

Hints and Tips:
SECPROD (RACF initialization)
RACF needs to read RACLISTed profiles from the DB at initialization
Use of RACGLIST can speed up loading of this information
Can be delayed if other RACFs have serialized the DB data sets

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
0
3
-
2
0
4
®
What lessons can we learn?

There MAY be significant savings to be achieved, IF you happen
to have times close to the maximum values
Most of these involved WTORs, so could be addressed by removing the
reason why the WTOR is issued (better systems management)
If your times are close to the median, addressing an individual
activity is unlikely to deliver significant savings:
Changing the BLKSIZE of your LNKLST data sets probably will not
earn you a 7-figure bonus....
But if you can address something that impacts MANY of these
activities, the savings may be more noticeable:
If you could move your system data sets to faster disks, that might
make a worthwhile difference


®
IPL Sequence
Hardware IPL
IPL RIMs
NIP RIMs
Now that the system is

MSI Part 1
up, we need to start
vital infrastructure
MSI Part 2
address spaces: SMS, IEACMD00 COMMND00
JES, LLA, VLF, VTAM,

Automation
TCP, OMVS
JES2
OMVS
SMS
VTAM/TCP

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
0
5
-
2
0
6
®
Starting the infrastructure

We carried out multiple tests, starting these address spaces as
SUB=MSTR, moving the START commands in and out of
COMMNDxx, and even deliberately delaying the start of VLF
and LLA (in case they were delaying the other "more important"
subsystems).
Using SUB=MSTR had negligible impact on the elapsed time between
IPL and when the subsystem started.
The restrictions associated with being a master subsystem were more
trouble than any small benefit that might have been achieved.


®
SMS considerations
Even though SMS is not started by the system (you must define
it in the IEFSSNxx member), its initialization time is included in
IPLSTATS:
SMS should be the first subsystem defined in IEFSSNxx
During initialization, SMS has to retrieve the SMS objects from the
ACDS - however this is typically a small part of the initialization time
unless you have thousands of objects
SMS also builds control blocks for every volume in every storage group
- for a large number of volumes, this could be very time-consuming -
APAR OA23197 introduced a more efficient algorithm
It also makes some updates to the ACDS, getting a RESERVE to
protect each update

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
0
7
-
2
0
8
®
SMS considerations
Watch for "waiting for SMS subsystem" type messages during
IPL to see if SMS is delaying other subsystems
12:46:16.74 00000010 *IGW456I SMSVSAM ADDRESS SPACE INITIALIZATION IS 127
127 00000010 WAITING FOR SMS ADDRESS SPACE TO BE MARKED AVAILABLE.
12:46:18.48 00000010 IGD020I SMS IS NOW ACTIVE
12:46:18.75 00000010 IGW456I SMSVSAM ADDRESS SPACE INITIALIZATION IS 286
286 00000010 RESUMING AFTER SMS ADDRESS SPACE IS MARKED AVAILABLE.
But notice the time difference between the IGW456I messages

- if that were large, this could be a source for concern, but in
this case, is not a problem
May also see IGD033I message "JOB jobname IS WAITING
FOR THE SMS ADDRESS SPACE TO INITIALIZE"


®
JES2 considerations
Apparently some customers avoid IPLing multiple systems
concurrently because of the JES2 contention messages.
These messages do not indicate a problem - they are simply JES' way
of letting you know why it is waiting
You should NOT let these messages stop you from IPLing multiple systems
concurrently
Some general recommendations:
Place JES2 checkpoint in CF for faster access
Avoid huge numbers of data sets on JES2 PROC statements
Do NOT use "*" in range subscripts in JES2 parms
Make sure init parms are clean - do NOT want operators to receive a WTOR
Enable automatic restart processing (using AUTOEMEM & RESTART=YES)
When you stop a JES, other members of the plex can do processing that would
otherwise need to be done when that member restarts

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
0
9
-
2
1
0
®
JES2 considerations
To obtain an insight into where the time is being spent during
JES2 initialization, use the $DPERFDATA command
$DPERFDATA
$HASP660 $DPERFDATA 940
$HASP660 STATISTICS FROM INITIALIZATION:
This command is not

$HASP660 ROUTINE=MVSSTART,TIME=2.538686,CPU=0.002041,
$HASP660 ROUTINE=LOADINIT,TIME=0.438797,CPU=0.001899,
$HASP660 ROUTINE=IRMODCHK,TIME=0.460242,CPU=0.002105, documented, but see
RedPaper JES2
$HASP660 ROUTINE=IRSSI,TIME=1.498963,CPU=0.042562,
$HASP660 ROUTINE=IROPTS,TIME=0.036047,CPU=0.000734,
$HASP660 ROUTINE=IRSETUP,TIME=0.068792,CPU=0.017270, Performance and
$HASP660 ROUTINE=IRENF,TIME=0.000102,CPU=0.000100,
$HASP660 ROUTINE=IRPL,TIME=0.030373,CPU=0.018666, Availability
$HASP660 ROUTINE=IRPOSTPL,TIME=0.163773,CPU=0.004072, Considerations,
$HASP660 ROUTINE=IRDCTDCB,TIME=0.000361,CPU=0.000277,
$HASP660 ROUTINE=IRURDEV,TIME=0.000001,CPU=0.000001, REDP3940 for more
$HASP660 ROUTINE=IREMVS,TIME=0.013006,CPU=0.012037,
$HASP660 ROUTINE=IRDA,TIME=1.207206,CPU=0.078524,
info
$HASP660 ROUTINE=IRNJE,TIME=0.053006,CPU=0.048953,
$HASP660 ROUTINE=IRRJE,TIME=0.002341,CPU=0.002124,
$HASP660 ROUTINE=IRCSA,TIME=0.000209,CPU=0.000193,
The upcoming MTTR Redbook will provide more information about some of
these fields


®
OMVS initialization
OMVS is required for many z/OS components, including those
that are prerequisites for other important components (TCP, for
example).
If OMVS initialization is delaying other address spaces, you may see
messages such as "EZZ9314E TCP/IP WAITING FOR OMVS TO
INITIALIZE"

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
1
1
-
2
1
2
OMVS initialization
When using UNIX File System Sysplex File Sharing, you must
have an OMVS CDS. Like the Sysplex CDS, the size of the
OMVS CDS depends on the values you use when you format it.
The following results are from some tests we ran:
CDS Size Time to "ZFS start" Time to "OMVS
Initialization Complete"
32 6 17
1336 7 34
1336 6 35
These indicate that the size of the CDS does not impact OMVS
initial processing, however it DOES impact mount time (not just
at IPL, but for every mount)
Remember that DECREASING the size of a CDS requires a sysplex
IPL, but if you are increasing the CDS, don't oversize it.
OMVS Initialization
Sysplex file sharing implements a function shipping model for
filesystems that are mounted R/W, where both write AND read
requests are routed to whichever system currently owns the file
system (via XCF)
If possible, move all r/w files out of the root file system so
that it can be mounted R/O on all systems (and avoiding the
function shipping cost)
z/OS R11 includes performance enhancements to sysplex file
sharing

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
1
3
-
2
1
4
®
OMVS Initialization
Objective is to get OMVS initialized as quickly as possible, so
recommend to only specify file systems that are absolutely
required in the BPXPRMxx member
Mount additional file systems later using automation
Or, could define filesystems as AUTOMOUNT and run a batch job to
touch those file systems after the IPL has completed
Also, review the contents of /etc/rc to ensure it only contains
commands that MUST be issued as part of IPL


®
OMVS Initialization
It is very important to do an orderly shutdown of OMVS:
AFTER you stop any subsystem that might be writing to the file
system, including TCP
Issue F OMVS,SHUTDOWN command to stop OMVS
Especially for zFS, the recovery processing can be significant
during the next IPL if OMVS is not stopped cleanly
If you wish, you can stop zFS separately using the F
OMVS,STOPPFS=ZFS command.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
1
5
-
2
1
6
®
TCP and VTAM

The "IPL Completed" time at the end of the IPLSTATS report
reflects the completion time of the initialization of the TCP
stack.
TCP initialization can't start until VTAM finishes processing
VTAMLST
There are (nearly) no known things you can do to make TCP take
very long to start:
One exception is if you have a very large number of VPN tunnels to
open
However, TCP can't start until OMVS has initialized, so you should
ensure there are no delays impacting OMVS.


®
VTAM
VTAM initialization is generally very short
However, be wary of spreading VTAMLST definitions over many
members as this can result in a large number of I/Os required to
retrieve all the VTAM definitions
Some subsystems that used to REQUIRE that VTAM initialization had
completed before they could be started (CICS, for example) will now
keep retrying if VTAM is not available when they expect it to be - this
means that those subsystems can now be started earlier in the IPL
process

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
1
7
-
2
1
8
®
LLA and VLF

The cost of starting LLA and VLF is small, so there is no
benefit in delaying their startup.
On the contrary, the buffering function they provide may help
deliver benefits to the time to load required programs for all
other components.
Both VLF and LLA are started with SUB=MSTR - while this is
not really necessary, there is no reason not to do so.


®
Outstanding issues
Still working on the following (all of which should be addressed in
the Redbook):
The most efficient way of handling SMF Type 30 records, especially in
tandem with very large DB2s (10s of thousands of data sets open).
We want to measure to see if the use of SMF Logger would have any
impact on DB2 shutdown time.
Ensure we have covered all the considerations for OMVS.
Collate the results for all the subsystem changes we tested

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
1
9
-
2
2
0
®
Where to start your MTTR project?

First step is to understand and track where you are today
Even if MTTR is not an issue today, you should monitor it to enable you
to proactively address any future problems
Address systems management issues
You don't want to waste your time tuning a process if the whole
process is not needed
Identify the "big hitters"
There is not much point tuning a process that takes 20 ms.
The elapsed time for elements of the IPL vary from one customer to
another, so you want to address the elements that most impact you


®
Finding your big hitters

There are some tools you can use to investigate your
startup and shutdown times:
SYSLOG - Helpful for all areas of investigation, but especially
for early stages of IPL (when there is no RMF running).
JOBLOG - May provide more information than syslog.
IPLSTATS - Program to format the IPST control block. This
contains information about elapsed time for about 125 steps of
z/OS IPL.
Can be formatted with IPCS command or IPLSTATS program
Syslog analysis tool and IPL comparison tools:
MSGLG610 and IPLMERG4
Tools available from IBM on an as-is basis
Various SMF records.

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
2
1
-
2
2
2
®
Some helpful jobs and programs...

To write IPST values to syslog (for later capture):
//IPLSTATS PROC
//IPLSTATS EXEC PGM=IPLSTATZ
//STEPLIB DD DSN=KYNEF.IPLSTATS.LOAD,DISP=SHR
To write the IPST values directly to a data set:

//IPLSTATX PROC
//IPLSTATX EXEC PGM=IPLSTATX
//STEPLIB DD DSN=KYNEF.IPLSTATS.LOAD,DISP=SHR
//OUTPUT DD DISP=(,CATLG),
// DSN=ZS9037.ZIPLSTAT.&SYSNAME.(+1),
// LIKE=ZS9037.ZIPLSTAT.&SYSNAME..MODEL,
// SPACE=(TRK,(1,1)),
// UNIT=SYSDA
Both programs will be provided as Additional Material in the

MTTR Redbook
Contact me for a small REXX exec to translate IPSTATS report into
CSV format

®
IPLSTATS Disclaimer
The information provided in the IPLSTATS report is not a
formal IBM published interface, and the IPLSTATS program is
provided on an as-is basis. You may find that all the times do
not add up - this is because not every module that executes
during the IPL process is reported on. However, the information
provided for each process/module is reliable and provides an
invaluable insight into where the time is being spent during IPL
that you would not otherwise have

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
2
3
-
2
2
4
®
Some helpful jobs...

To capture a subset of syslog - run SDSF in batch:
//SDSFBAT EXEC PGM=SDSF
//LOGOUT DD DSN=ZS9037.RUNFK01.SYSLOG.DISP=OLD
//ISFOUT DD SYSOUT=*
//ISFIN DD *
LOG S
SYSID #@$A
PRINT FILE LOGOUT
PRINT 12.10.00 08/13/09 14.45.00 08/13/09
PRINT CLOSE
/*
You need special permissions to run SDSF in batch. If you are using
ISFPARMS for security, an additional group for batch is needed (see SDSF
book).
However, it is easier to use SAF. If this is a test system, define a profile
in the SDSF class:
rdefine sdsf group.** uacc(read)
That will map all users to group 1, which has LOG authority.

®

To analyze syslog (and create the input for the IPLMERG
program): //MSGRATE EXEC PGM=MSGLG610,REGION=5000K
//OPTIN DD *
TITLE: ITSO MTTR RESIDENCY
OFFSET(1)
REPORT(SUM,AMSG,CMSG)
IPLSTATS(LIST)
IPLID(FK08)
YEAR4
/*
//DATA DD DSN=ZS9037.RUNFK91.SYSLOG,DISP=SHR
//TTLLIB DD DSN=ZS9037.MSGANAL.TEXT,DISP=SHR
//SYSUDUMP DD DUMMY
//SYSPRINT DD DUMMY
//IPLST DD SYSOUT=*
//IPLSQ DD DSN=ZS9037.RUNFK01.IPLSTATS,DISP=OLD
//PRNTOUT DD DUMMY
//COMPRATE DD DUMMY
//COMPMSG DD DUMMY
//UNKNOWN DD DUMMY
//PREVIEW DD DUMMY
//IMSGRATE DD DUMMY
//BURST DD DUMMY
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
2
5
-
2
2
6
®

To compare the elapsed time of two IPLs:
//IPLMERG EXEC PGM=IPLMERG4,REGION=5000K
//STEPLIB DD DSN=KYNEF.MSGLGPGM.LOAD,DISP=SHR
//INPUT1 DD DISP=SHR,DSN=ZS9037.RUNFK01.IPLSTATS
//INPUT2 DD DISP=SHR,DSN=ZS9037.RUNFK02.IPLSTATS
//OUTPUT DD SYSOUT=*
//OUTPUT1 DD SYSOUT=*
//OUTPUT2 DD SYSOUT=*
IPLMERG and MSGLG610 programs are developed by Kevin Kelley

and will hopefully be available on z/OS Tools and Toys Web site
soon
These programs are not foolproof, but they are a LOT better than
trying to analyze all this stuff manually.


®
Further information
Upcoming Redbook - System z Mean Time to Recovery Best
Practices, SG24-7816
Presentation entitled z/OS MVS System Initialization Logic -
Initial Program Load (IPL) by Silvio Sasso, on Techdocs at:
http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS3699
z/OS Planned Outage Avoidance Checklist, SG24-7328

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
2
7
-
2
2
8
򔻐򗗠򙳰
ibm.com/redbooks Infiniband links - recap and performance
Redbooks Workshop
Infiniband
New CF link type (CIB) for all IB coupling links
System z9 (with limitations)
System z10
7 subchannels per CHPID applies to CIB links
Multiple CIB CHPIDs per physical IB Coupling link
A single physical link can “share” CHPIDs across multiple CF
images, same or different sysplexes
Can provide additional subchannels without additional physical
links (by assigning multiple CHPIDs per link)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
2
9
-
2
3
0
®
Infiniband
Up to 16 CHPIDs using same physical links
More subchannels / physical link
NOT more subchannels / CHPID
Can connect to multiple CF LPARs
CF Receiver CHPIDs can share link
z/OS
CHPID FF
7 subchannels Single PSIFB link
CHPID FE 14 subchannels
7 subchannels


®
Infiniband
If need more subchannels, can define more CHPIDs to a physical
link to get them
More subchannels can be beneficial for long links as can get
more bandwidth
Can have more CF’s on a box if links were limiting factor
Fewer physical links implies easier to configure (less cabling)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
3
1
-
2
3
2
®
Maximum links per server
Server IC IFB ICB-4 ICB-3 ICB ISC-3 Max # Links

z800 32 - - 5 - 24 26 + 32
6 (0CF)
z900-100 32 - - 16 16 32 64
CF 42 w/RPQ
z900 32 - - 16 8 32 64
16 w/RPQ
z890 32 - 8 16 - 48 64
z990 32 - 16 16 8 48 64
z9 EC 32 16 16 16 - 48 Peer 64
z9 BC 32 12 16 16 - 48 Peer 64
z10 EC 32 32 16 - - 48 Peer 64
32 IFB + ICB-4
z10 BC 32 12 12 - - 48 Peer 64
56 External
12 IFB + ICB-4


®
Supported PSIFB connectivity
z/OS z890 / z9 z10

CF z990
z890 / z990
N/A N/A N/A
z9
N/A No Yes
Standalone CF only
z9
N/A No Yes
Mixed ICF & z/OS
z10
N/A Yes Yes

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
3
3
-
2
3
4
Maximum supported link distances
Distance IC ICB-4 12x IFB ISC-3

1x IFB
Within server Yes Yes Yes Yes
<10 m Yes Yes Yes
10 m – 150 m Yes Yes
150 m – 100+ km Yes

Comparison of ISC to 12x PSIFB links
ISC PSIFB
LOCK1 Total Activity Rate 15130.11 LOCK1 Total Activity Rate 16055.27
LOCK1 Sync Activity Rate 2082.71 LOCK1 Sync Activity Rate 15875.84
LOCK1 Sync ST 26.23 LOCK1 Sync ST 13.47
LOCK1 Async Activity Rate 13047.40 LOCK1 Async Activity Rate 179.43
LOCK1 Async ST 119.82 LOCK1 Async ST 84.84
GBP3 Total Activity Rate 11507.44 GBP3 Total Activity Rate 13419.46
GBP3 Sync Activity Rate 60.12 GBP3 Sync Activity Rate 10024.09
GBP3 Sync ST 63.44 GBP3 Sync ST 16.93
GBP3 Async Activity Rate 11447.32 GBP3 Async Activity Rate 3395.38
GBP3 Async ST 176.05 GBP3 Async ST 142.01

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
3
5
-
2
3
6
Comparison of ICB4 to 12x PSIFB links
ICB PSIFB
LOCK1 Total Activity Rate 15643.26 LOCK1 Total Activity Rate 16055.27
LOCK1 Sync Activity Rate 15350.19 LOCK1 Sync Activity Rate 15875.84
LOCK1 Sync ST 10.57 LOCK1 Sync ST 13.47
LOCK1 Async Activity Rate 293.07 LOCK1 Async Activity Rate 179.43
LOCK1 Async ST 88.13 LOCK1 Async ST 84.84
GBP3 Total Activity Rate 13006.05 GBP3 Total Activity Rate 13419.46
GBP3 Sync Activity Rate 12475.26 GBP3 Sync Activity Rate 10024.09
GBP3 Sync ST 14.84 GBP3 Sync ST 16.93
GBP3 Async Activity Rate 530.79 GBP3 Async Activity Rate 3395.38
GBP3 Async ST 203.33 GBP3 Async ST 142.01

Comparative performances
ISC ICB4 PSIFB 12x
Transaction Rate – 146.61 149.86 149.20

ALL Txns
Response Times 0.052 0.028 0.031
TOT - ALL Txns
EXEC - ALL Txns
Transaction Rate – 25.43 26.18 25.80

TPNO Txns
TOT – TPNO Txns
EXEC – TPNO Txns
CPU Utilization (% of 116.74 99.72 106.57

1 CP)
Capture Ratio .92 .95 .94

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
3
7
-
2
3
8
®
Infiniband response times

At the time the previous set of measurements were taken, we
did not have 1x Infiniband links available - we now have both
12x and 1x Infiniband links on our z10
To get a feel for the relative performance (NOT capacity or

bandwidth) of 1x links compared to 12x links, we ran an
artificial workload with both link types
Configuration was:
1 z/OS 1.10 LPAR on z10 with 3 dedicated CPs
1 CF LPAR on z10 with 1 dedicated ICF
2 dedicated PSIFB links (either 12x or 1x) to CF

®
CF CPU util and Req rate

10 10000
9 9000
8 8000
7 7000
6 6000 CF CPU Util
5 5000
Req/Sec
4 4000
3 3000
2 2000
1 1000
0 0
ICP 1x PSIFB 12x PSIFB

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
3
9
-
2
4
0
Response times by link type

100
90
80
70
60
Sync RT
50
Async RT
40
30
20
10
0
ICP 1x PSIFB 12x PSIFB

򔻐򗗠򙳰
ibm.com/redbooks Availability enhancements
Redbooks Workshop

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
4
1
-
2
4
2
®
ALLOCxx enhancement
Back in 2006 we did a Redbook about Planned Outage Avoidance
- mainly about things that can be changed dynamically that
previously required an IPL.
z/OS R11 introduces the ability to make dynamic changes to the

values you specify in the ALLOCxx member.


®
ALLOCxx
The ALLOC member lets you control defaults for various
allocation-related values and actions:
SPACE
UNIT
TIOT
SDSN_WAIT
VOLUME_ENQ
VOLUME_MNT
SPEC_WAIT
ALLC_OFFLN
CATLG_ERR
VERIFY_VOL
2DGT_EXPDT
SYSTEM (new in z/OS R11)
D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
4
3
-
2
4
4
®
Changes in R11
z/OS R11 adds two new commands:
Display ALLOC,GRPLOCKS|OPTIONS
SETALLOC (note that there is no SET ALLOC=xx)
Supports changing any of the ALLOC parms EXCEPT 2DGT_EXPDT
Notice small difference in syntax between ALLOCxx member and
SETALLOC command....
Member uses (xx) format
Command uses =xx format


®
D ALLOC commands

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
4
5
-
2
4
6
®
SETALLOC command


®
ALLOC enhancements in z/OS R11

DISPLAY ALLOC + SETALLOC = 1 less reason to IPL
Note that support is NOT rolled back to earlier releases of

z/OS

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
4
7
-
2
4
8
򔻐򗗠򙳰
ibm.com/redbooks Latest XCF signalling performance numbers
Redbooks Workshop
XCF Signalling performance
XCF Signalling Rates

100
90
80
70
Thousands
60 FICON EX4
50 FICON EX8
40 ISC3
30 ICB4
20 IB 12X
10
0
Req/Sec (1K)

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
4
9
-
2
5
0
XCF Signalling performance
XCF Signalling Response Times

XCF I/O Response Time (msec)
1
0.75
0.5 FICON EX4

FICON EX8
ISC3
0.25 ICB4
IB 12X
0
1K 8K 32K
XCF Message Size

򔻐򗗠򙳰
ibm.com/redbooks Miscellaneous, odds and ends, bits and pieces....
Redbooks Workshop

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
5
1
-
2
5
2
®
Interesting Web sites

John Kettner in IBM puts out frequent short technical articles
about all sorts of interesting topics. These are hosted by
Marist college in Poughkeepsie. For the list of articles, go to:
http://www.idcp.org/learnzos
Web site containing z/OS 1.11 manuals as well as all Hot Topics
newsletters:
http://www.ibm.com/systems/z/os/zos/bkserv/r11pdf/
SHARE user conference presentations are available to the public:
http://www.share.org (click on Proceedings)


®
Interesting tools
Have you ever done a search in SDSF for something that
appears a few lines BEFORE some fixed text that you can
search on?
Do you end up doing repetitive PF5s then PF7s to go back a screen to
see what you are really looking for?
You can use the EDITSET ISPF option in SDSF (SE) to set the current
line - so instead of the found line being line 1 on your screen, you can
say that you want it to be line 5, so you can see the 4 lines prior to the
found line without having to PF7 back...

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
5
3
-
2
5
4
®


®
Interesting APARs
OA26037 - Toleration APAR to let z/OS R9 and R10 understand a CDS
formatted to support System Status Detection function
OA26993 and OA26995 - Enable AutoIPL when used in conjunction
with z/OS R10. OA30284 for z/OS R11 still open at time of writing
OA28603 - Adds reporting of the XES Heuristic thresholds to the
output from the D XCF,C command.
PK83260 - A new parameter, NODSEXIST, has been added to the
DSF INIT command to not allow volumes which contain data sets other
than the index data set and/or VVDS to be initialized. THIS SHOULD
BE ADDED TO ALL YOUR DSF INIT JOBS.
OA27945 - Rolls zAAP on zIIP capability back to z/OS 1.9
OA25559 - Adds Solid State Disk support to DS8000
OA26084 and OA29017 - High Performance Ficon support

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
5
5
-
2
5
6
®
Questions?


®
Thanks!!
And please remember to complete your evaluations

D
a
t
e
:
1
1
/
0
5
/
0
9
P
a
g
e
2
5
7
-
2
5
8

ITSO Parallel Sysplex & High Availability Topics PDF

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

ITSO Parallel Sysplex & High Availability Topics PDF

Загружено:

Авторское право:

Доступные форматы

򔻐򗗠򙳰 ®

International Technical Support Organization

Parallel Sysplex and High Availability

IBM ITSO - International Technical Support Organization

© 2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰 ®

®2009 IBM Corporation. All rights reserved.

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

®2009 IBM Corporation. All rights reserved.

Using the term Redbooks in written text

Correct usage for written text :

Using the logo:

OTHER ITSO PUBLICATIONS - Marks not yet registered

The following terms are trademarks of other companies:

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰 ®

The following terms are trademarks of other companies:

CA is a trademark of Computer Associates

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks z/OS 1.11 Sysplex-related enhancements

© 2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰 ®

z/OS 1.11 Enhancements for sysplex

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks Base Control Program internal interface

© 2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰 ®

®2009 IBM Corporation. All rights reserved.

BCPii interface as opposed to BCPii interface...

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

®2009 IBM Corporation. All rights reserved.

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

Who would use BCPii?

®2009 IBM Corporation. All rights reserved.

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

First step is to give LPARs authority to issue commands to other LPARs...

®2009 IBM Corporation. All rights reserved.

You are logged

Select the CPC

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

Select Save and

®2009 IBM Corporation. All rights reserved.

This may take a

Recommend verifying that

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

If the activation profiles were NOT updated......

You are logged

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

This option must be

®2009 IBM Corporation. All rights reserved.

Setting up BCPii - HW end

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰

®2009 IBM Corporation. All rights reserved.

Then click on Add in

®2009 IBM Corporation. All rights reserved.

ibm.com/redbooks International Technical Support Organization 򔻐򗗠򙳰