Partitioned Data Set Extended (PDSE) Hiperspace (Caching) Problems, Considerations, and Parameter Changes

Abstract
Users of Partitioned Data Set Extended (PDSE) data sets may experience high CPU utilization in the SMSPDSE address space. This has caused a problem for many users. For related information about this topic, refer to the following IBM Redbooks publication: Partitioned Data Set Extended Usage Guide, SG24-6106-01

Contents
The high CPU utilization problem accessing PDSEs is caused by the unexpected caching of a large volume of program objects. This high CPU utilization is not a problem for those PDSE users who have a Hiperspace size specified or for those users whose applications fit the caching model. This technote provides some background, discusses the problem, and presents the solution.

PDSE Hiperspace (caching) considerations: background

PDSE Hiperspace represents a caching function implemented to improve PDSE performance in those cases where the same member (or members) are accessed repeatedly. System storage and the CPU cycles required for caching replace the overhead of repeatedly reading members from DASD. As a result, benefits obtained from using PDSE Hiperspace depends on the application and its PDSE member access patterns. (PDSE directories are automatically cached by the buffer management facility and are not part of this discussion.) Heavily used PDSEs and those that stay open for long periods of time are the best candidates for PDSE Hiperspace caching. Be sure your application fits the caching model before putting the PDSE into "must cache" storage class. PDSE system data sets (for example, those in LNKLST) are usually managed by LLA/VLF. Any program object that is cached in Hiperspace and at the same time is also managed by LLA/VLF will, in time, be dropped from the Hiperspace, so it is better to prevent these program objects from going into the Hiperspace in the first place. For SMS-managed PDSE data sets, the use of Hiperspace caching is controlled by the Millisecond Response Time (MSR) value in the associated Storage Class. The MSR values apply not just to PDSE caching, but have wider applicability to the DASD storage units. In the case of PDSE data sets, only an MSR value that translates to "must cache" causes member caching. MSR values that translate to "may cache" and "never cache" will not cause members to be cached. For example, a storage class with MSR set to a value of 1 indicates "must cache" and 999 means "never cache."

(Refer to the section below titled Millisecond Response Time for more information about MSR values and caching.) Ensure that SMS-managed PDSEs are associated with storage classes that have appropriate MSR settings. (Note that PDSE data sets shipped as part of the operating system are generally not SMS-managed, so they have no storage class and therefore cannot be cached). If the PDSE Hiperspace is to be used, then its use of CPU cycles, real storage, or both must be balanced against the benefit obtained. Hiperspaces use system storage (real and virtual) and must be tuned to avoid excessive use at the expense of other system functions. Hiperspace pages are the last to be stolen in a real storageconstrained environment, so the size of the Hiperspace may have to be limited when real storage is constrained. In addition, there has always been a tradeoff between CPU and I/O costs. Caching reduces I/O path and increases CPU utilization. It is a tradeoff made by the installation and applications depending on the access patterns for that particular data set.

Recent APAR history that created and solved several PDSE Hiperspace caching problems
Summary of problem and solution: The recent problems with CPU utilization resulted from fixing a PDSE bug, which fixed PDSE Hiperspace caching. As the default it abruptly started working again for almost all PDSE users. PDSE Caching Features - original design. (PDSE caching limited by ESA Architecture)

1. For the installation to enable PDSE caching at that time, there were two requirements: a. In the IGDSMSXX PARMLIB parameter, HSP_SIZE had to be a non-zero value.
The product was shipped with a default size of 256 meg. b. In addition, the customer had to have Expanded Store, which had to be bought separately. c. Few customers actually bought Expanded Store (and probably not any small customers), so PDSE caching was not common. 2. In addition, PDSE caching could (and can) be controlled at the data set level by Storage Class. Even though caching was enabled at the installation level, it could be regulated by Storage Class MSR value. a. It is believed that few customers took advantage of this Storage Class feature in the beginning.

z/Architecture was introduced. This removed the limiting factor that ESA Architecture had, such as the need for Expanded Store.

1. Expanded store was replaced by real storage in the new processors. All customers
automatically had it. PDSE code was changed, via OW53245, to recognize and use the real storage for a Hiperspace. 2. This had the effect of enabling all installations with z/Architecture to PDSE caching, since the default of 256 meg remained in effect. 3. Whereas, in the past the customer had to do something explicit to get caching (such as buy Expanded Store), it now came automatically.

Yet PDSE caching continued to be limited, blocked by a bug.

1. A bug prevented any pages from being thrown away (that is, they never aged out). When the cache was full, further caching was blocked and caching stopped. This bug was fixed with OA06884, which was released 06/07/04. This fix had the effect of abruptly enabling caching for many customers. 2. Soon after the release of this APAR, it was obvious that it was causing a problem with CPU usage. There were two attempts to reissue the fix with further HOLD information to warn the customers (via PE APARs OA08991, released 10/20/04, and finally OA10323, released 03/04/05).

In addition, Storage Class stopped controlling caching at some point. 1. Another bug caused Contents Supervisor to ignore Storage Class and always cache, so when OA06884 was released all program objects were cached. This was fixed via APAR OA09162, released 01/13/05.

To resolve the PDSE issue with CPU utilization 1. APAR OA11068 (target release 06/15/05) changes the default to not enable caching for z/Architecture, by changing the default to HSP_SIZE=0 for z/Architecture systems. a. The installation can enable caching by changing the HSP_SIZE= to a non-zero value. This requires an IPL for all releases except z/OS V1.6, in which it is possible to restart the PDSE restartable address space if this feature is enabled. Note that for z/OS V1.6 each of the two address spaces, SMSPDSE and SMSPDSE1, need their own specification for HSP_SIZE. In all cases the specification of HSP_SIZE requires a re-IPL except for the restartable address space (SMSPDSE1), whose HSP_SIZE can be changed by restarting the address space. 2. Storage Class continues to be available to control caching at a PDSE level (via the MSR parameter). 3. The maximum value of MSR=999 means NEVER CACHE. Decreasing values mean MAYBE CACHE or MUST CACHE, subject to the DASD control unit characteristics. MSR=1 ensures MUST CACHE. (MAYBE CACHE is treated as NEVER CACHE for PDSEs). This decision is based on how the data set is accessed. 4. If an application does a lot of opens and closes, the PDSE should not be cached. Members are cached only while the data set is open by at least one application on the system. 5. Caching works best for PDSEs that are open a long time with frequent references to the same members.

APAR OA11068 also modifies the Least Recently Used (LRU) default values. 1. LRU values can be modified by the installation using the SETSMS command. This does not take an IPL. These values can be changed dynamically. 2. LRUTIME( ) specifies the number of seconds between running the LRU queue. LRUCYCLES( ) specifies how many times LRU must run before throwing out a page. a. Default LRU values shipped with the product in the past were LRUTIME(15), LRUCYCLES(240). These defaults will remain for non-z/Architecture. For z/Architecture, however, this may be too often to search and too long to keep a page in cache. b. For z/Architecture default LRU values will be changed to LRUTIME(60), LRUCYCLES(15). This will cause the code to run the LRU queue once per minute,

keeping pages in cache for 15 minutes before throwing them out. This should control some of the possible thrashing and reduce the frequency of searching the queue, thus reducing the CPU utilization. 3. The following sample SETSMS commands can be used to change the LRUCYCLES and LRUTIME values: SETSMS PDSE_LRUCYCLES(nnn) SETSMS PDSE1_LRUCYCLES(nnn) SETSMS PDSE_LRUTIME(nn) SETSMS PDSE1_LRUTIME(nn) 4. Customers can tune their systems for performance by tweaking these LRU values. SMF provides caching statistics by Storage Class. 5. SMF records can provide a list of Storage Classes that are not caching well, such as those where the hit ratio is low.

This APAR (OA11068) is marked HIPER for performance reasons with a ++HOLD ACTION and + +HOLD DOC because default parameters are being changed. It will prereq the PTFs associated with APAR OA09162 (Storage Class) and APAR OA10323 (fixes caching by throwing out aged pages). Target for release of this PTF is 6/15/05. When this is installed, CPU utilization tradeoffs will be under the control of the installation.

Reference notes
The following additional information relates to PDSEs and caching for your reference.

Millisecond response time

The millisecond response time (MSR) serves two purposes in SMS: First, it is used as the performance objective for selecting candidate volumes for new data set placement. During a new data set allocation, SMS searches for a volume that meets or closely matches this objective. If no volume satisfies the objective, then SMS attempts to find a volume that comes closest to matching it. If more than one MSR is explicitly or implicitly specified, the storage class and associated device MSRs are averaged and compared. Second, if the data is placed on a volume attached through an IBM 3990 Storage Control with cache, and cache is enabled for that volume, the MSR is used to determine whether caching is mandatory, optional, or should be inhibited for the data set. This attribute does not apply to objects. The PDSE implementation of MSR is different from the SMS implementation. PDSE data sets use the PDSE Hiperspaces for caching; the Hiperspace caching is either must cache or never cache. For PDSE data sets, specify an MSR value of 1 for must cache. Specify an MSR value of 999 for never cache. Refer to "Defining Performance Objectives" in z/OS DFSMSdfp Storage Administration Reference, SC26-7402, for detailed information about the affect of MSR on different storage device types and models.

Optional keywords for IGDSMSxx

HSP_SIZE | PDSE_HSP_SIZE({nnn})

Default: 0 for z/Architecture mode, 256 for non-z/Architecture mode With the introduction of a second PDSE address space in z/OS 1.6, there are now two sets of PDSE Hiperspace parameters. For z/OS V1.6 and greater, HSP_SIZE is synonymous with PDSE_HSP_SIZE and is associated with the SMSPDSE address space. HSP_SIZE specifies the size of the Hiperspace in megabytes that is used for PDSE member caching for SMSPDSE. You can use the HSP_SIZE parameter to request up to 2047 MB for the PDSE Hiperspace. You can indicate that the Hiperspace is not to be created by setting HSP_SIZE to 0. If the Hiperspace is not created, the system will not cache PDSE members. On systems that are running in z/Architecture mode, when OA11068 is installed, by default the PDSE Hiperspace is not created and PDSE member caching is disabled. On non-z/Architecture mode systems that have expanded storage, by default the PDSE Hiperspace uses either 256 MB of expanded storage or one-half of the system's available expanded storage, whichever amount is lower. If no expanded storage is online to the system, the Hiperspace cannot be created and PDSE member caching is disabled. If you specify a valid value for HSP_SIZE, the system uses it to create the PDSE Hiperspace at IPL time. The HSP_SIZE value remains in effect for the duration of the IPL. If there is not enough appropriate storage available to satisfy the HSP_SIZE value, the system uses some portion of the available storage (up to the full amount) for the PDSE Hiperspace depending on the amount of caching activity in the system. The system stops caching PDSE members if the available storage becomes full. Use the HSP_SIZE parameter with care. If you specify an HSP_SIZE value that is too low for normal PDSE Hiperspace usage, you can degrade PDSE performance. If you specify a value for HSP_SIZE that is too large, and there is contention for storage on the system, you can degrade performance of other components or applications in the system. To determine the current HSP_SIZE value of the PDSE Hiperspace, use the DISPLAY SMS,OPTIONS command, or review the messages that are written to syslog when SMS is started. You can examine SMF type 42, subtype 1, records to evaluate the effectiveness of a particular HSP_SIZE value.

LRUCYCLES | PDSE_LRUCYCLES(nnn) Default: 15 for z/Architecture mode, 240 for non-z/Architecture mode With the introduction of a second PDSE address space in z/OS V1.6, there are now two sets of PDSE parameters. For z/OS V1.6 and greater, LRUCYCLES is synonymous with PDSE_LRUCYCLES and is associated with the SMSPDSE address space.

PDSE_LRUCYCLES(nnn l 15)

Default: 15 PDSE_LRUCYCLES is associated with the restartable SMSPDSE1 address space. Refer to the RESTARTABLE_PDSE_AS(YES) parameter for information about the SMSPDSE1 address space. LRUCYCLES specifies the maximum number of times (5 to 240 cycles) that the least recently used (LRU) routine will pass over inactive buffers before making them available for reuse. While this parameter sets the maximum value, the LRU algorithm will dynamically change the actual number of times it passes over inactive buffers. LRUCYCLES is related to LRUTIME. A change to the LRUCYCLES value introduced by this parameter will take effect on the next execution of the LRU routine. Most installations should use the default value (changed by APAR OA11068, as mentioned above). In some very high data rate environments you may want to tune this value. You should monitor the SMF 42 type 1 record to determine the amount of caching activity in the BMF data space. See z/OS: MVS System Management Facilities (SMF), SA22 -7630, for information about the buffer management statistics recorded in SMF record type 42.

LRUTIME|PDSE_LRUTIME(nnn) Default: 60 for z/Architecture mode, 15 for non-z/Architecture mode With the introduction of a second PDSE address space in z/OS V1.6, there are now two sets of PDSE parameters. For z/OS 1.6 and greater, LRUTIME is synonymous with PDSE_LRUTIME and is associated with the SMSPDSE address space. PDSE1_LRUTIME is associated with the restartable SMSPDSE1 address space. Refer to the RESTARTABLE_PDSE_AS(YES) parameter for information on the SMSPDSE1 address space. LRUTIME specifies the number of seconds (5 to 60) that the system will wait between calls to the LRU routine. The LRU routine releases inactive buffers in the Hiperspace used to cache PDSE member data. LRUTIME is related to LRUCYCLES. A change to the LRUTIME value will take effect on the next execution of the LRU routine. Most installations should use the default value (which is changed by OA11068). In some very high data rate environments you may want to tune this value. You should monitor the SMF 42 type 1 record to determine amount of caching activity in the data space. See z/OS: MVS System Management Facilities (SMF), SA22-7630 for information about the buffer management statistics recorded in SMF record type 42.

BMFTIME|PDSE_BMFTIME(nnn) Default: 3600 With the introduction of a second PDSE address space in z/OS V1.6, there are now two sets of PDSE parameters. For z/OS V1.6 and greater, BMFTIME is synonymous with PDSE_BMFTIME and is associated with the SMSPDSE address space. PDSE1_BMFTIME is associated with the restartable SMSPDSE1 address space. Refer to the RESTARTABLE_PDSE_AS(YES) parameter for information about the SMSPDSE1 address space.

BMFTIME specifies the number of seconds that SMS is to wait between recording SMF records for buffer management facility (BMF) cache use. You can specify a value from 1 to 86399 (23 hours, 59 minutes, 59 seconds), and the default is 3600 (one hour). For information about the buffer management statistics recorded in SMF record type 42, see z/OS: MVS System Management Facilities (SMF) , SA22-7630.

PDSE1_HSP_SIZE({nnn|0}) Default: 0 With the introduction of a second PDSE address space in z/OS V1.6, there are now two sets of PDSE parameters. For z/OS 1.6 and greater, PDSE1_HSP_SIZE is associated with the restartable SMSPDSE1 address space. Refer to the RESTARTABLE_PDSE_AS(YES) parameter for information about the SMSPDSE1 address space. PDSE1_HSP_SIZE specifies the size of the Hiperspace in megabytes that is used for PDSE member caching for SMSPDSE1. You can use the PDSE1_HSP_SIZE parameter to request up to 2047 MB for the PDSE1 Hiperspace, or you can indicate that the Hiperspace is not to be created by setting PDSE1_HSP_SIZE to 0. If the Hiperspace is not created, the system will not cache PDSE members. On systems that are running in z/Architecture mode, by default the PDSE1 Hiperspace is not created and PDSE member caching is disabled. If you specify a valid value for PDSE1_HSP_SIZE, the system uses it to create the PDSE1 Hiperspace at IPL time. The PDSE1_HSP_SIZE value may be changed by a SET SMS=xx command; the new value will be used if the SMSPDSE1 address space is restarted. If not enough of the appropriate storage is available to satisfy the PDSE1_HSP_SIZE value, the system uses some portion of the available storage (up to the full amount) for the PDSE1 Hiperspace, depending on the amount of caching activity in the system. The system stops caching PDSE members if the available storage becomes full. Use the PDSE1_HSP_SIZE parameter with care. If you specify a PDSE1_HSP_SIZE value that is too low for normal PDSE Hiperspace usage, PDSE performance can degrade. If you specify a value for PDSE1_HSP_SIZE that is too large, and there is contention for storage on the system, it can degrade performance of other components or applications in the system. To determine the current PDSE1_HSP_SIZE value of the PDSE Hiperspace, use the DISPLAY SMS,OPTIONS command, or review the messages that are written to syslog when SMS is started. You can examine SMF type 42, subtype 1, records to evaluate the effectiveness of a particular PDSE1_HSP_SIZE value. SMF_TIME(YES|NO) Default: YES

SMF_TIME specifies whether DFSMS is to use SMF timing (that is, whether SMF type 42 records are to be created at the expiration of the SMF interval period, synchronized with SMF and RMF data intervals). The following SMF record 42 subtypes are affected when you specify SMF_TIME(YES): 1, 2, 15, 16, 17, 18. If you record these subtypes, you can use SMF_TIME(YES) to synchronize SMF type 42 data with SMF and RMF data intervals. Specifying SMF_TIME(YES) overrides the IGDSMSxx parameters CACHETIME and CF_TIME.