Performance Testing

1 Introduction
During performance testing projects, customers who do not require the full functionality of Compuware's ServerVantage monitoring product may still need to collect basic UNIX system activity data to enable bottlenecks to be found during analysis. These basic system statistics are similar to those which may be collected for Windows systems by configuring Remote Monitoring in Compuware's QALoad. This note discusses several possible ways forward: Option 1. Use SNMP. Collect statistics and store in the SNMP database (known as the MIB). As QALoad can be configured to collect values of SNMP variables (again via Remote Monitoring), this offers a good level of integration with QALoad's Analyze module. Option 2. Custom counters. Collect statistics which are added to QALoad's timing file by storing custom counters defined in a QALoad script. This would offer good integration but has the disadvantage of requiring scripting effort. Option 3. Collect statistics which are loaded into QALoad after a scenario has run. This note covers the following: - what statistics to collect, and what Solaris provides - configuring QALoad to collect values of these SNMP variables - identify any customisation required to collect missing values 1.1 Conclusions and recommendations

Regarding option 1, all the SNMP variables identified in this document can be monitored using Solaris SNMP agents. See section 2 for details. Unfortunately QALoad does not directly support collection of all of these variables, and cannot be configured to collect custom variables. As a result, Option 2 has been explored via a proof of concept, which indicates the approach is feasible, and this is the recommended approach to collecting missing data.

2 What statistics to collect, and what Solaris provides

2.1 What to collect As a baseline for the statistics that are required, the UNIX system resources implemented in Mercurys LoadRunner v8.0 were examined, and these are listed in a table below in columns 1 and 2. Articles on Mercurys support site confirm that LoadRunner does not address clustering scenarios in any special way. So we will assume when servers run in a livelive configuration, the requirement is for statistics on each to be collected separately and it should be possible to drill-down to each separately as well as viewing the overall picture.1
1

SNMP monitoring is described in the Sun Cluster 2.2 System Administration Guide, but this only covers state monitoring, and does not include any performance monitoring statistics.

Page 1 of 8

Performance Testing

LoadRunner collects statistics by making RPC calls directly to the UNIX rstatd(1M) daemon running in the system under test. The comparison table shows the names of the statistics provided by rstatd in column 3, and their data types in column 4. 2.2 What SNMP statistics Solaris provides SNMP is based around an extensible, hierarchical structure within which organisations can define SNMP variables to collect values they require. This is done by defining a custom MIB that may be installed in a run-time environment to allow data collection. SNMP traps are sent from an SNMP agent to a system management application when specified conditions are detected. This is designed for production system monitoring, and need not be enabled during performance testing. There are two different MIB definitions which have arisen independently which are relevant to Solaris system monitoring. These are listed in Appendix B of ref[6], and they are UCD-SNMP-MIB and the sun.mib (aka SUN-SEA-EXTENSIONS-MIB). 2.2.1 UCD-SNMP-MIB The Solaris 10 SNMP agent (snmpd) is known simply as the System Management Agent. Compared with previous versions of Solaris, it is a re-write derived from the Open Source project Net-SNMP (http://www.net-snmp.org), formerly known as ucdsnmp (as the package originated from UC Davis). The root of the sub-tree within the MIB is still named ucdavis. In Solaris 9, it would be possible to install Net-SNMP as a sub-agent alongside mibiisa (see next section) under control of the SNMP Master Agent. Variables collected by Net-SNMP are shown in columns 6 and 7 of the table below. 2.2.2 SUN-SEA-EXTENSIONS-MIB In Solaris 9 and earlier, a Sun specific MIB (sun.mib) was developed and supported by an SNMP agent known as mibiisa(1M). This was normally run as a sub-agent, for example running under control of the Sun Solstice Enterprise Master Agent (snmpdx(1M)). In Solaris 10, the Sun Solstice Enterprise Master Agent is replaced with the System Management Agent derived from Net-SNMP (see previous section). It is still possible to run mibiisa as a sub-agent if required, for example for backwards compatibility, but the recommended approach (ref [7]) is for SNMP requests for values in sun.mib to be sent to the seaExtensions extension module of the System Management Agent. This is an optional module it is important that it installed and configured to accept such requests. The sun.mib has also been renamed SUN-SEA-EXTENSIONS-MIB. Variables collected via the sun.mib are shown in column 5 of the table below. Page 2 of 8

Performance Testing

2.3

What rstatd statistics Solaris provides

The rstatd(1M) process was originally developed as part of SunOS (the precursor to Solaris) based around Suns RPC mechanism. So, the statistics which Mercury LoadRunner collects could be derived by making direct RPC calls. An investigation was done by building and testing a Linux version of rstatd(1M) downloaded from http://rstatd.sourceforge.net. This confirmed that the public domain version collects the statistics we require. As yet, no command line monitor has been located to issue the RPC calls. However, an hour of assembling code fragments from the Internet was sufficient to write a Linux command line monitor using the RPC stub code from the rstatd package. Providing an RPC library is available under Windows, it should be possible to port. Security concerns mean that RPC services (including portmapper) are often disabled on production systems. I would recommend SNMP is used in preference, unless other factors dictate that we reconsider the use of RPC.

Page 3 of 8

Performance Testing

2.4

Table comparing SNMP values to collect against what is provided

LoadRunner ref [1]
Average load

LoadRunner description
Average number of processes in Ready state during the last minute Collisions per second detected on the Ethernet Number of switches between processes or threads, per second Percent of time that the CPU is utilized Rate of disk transfers Errors per second while receiving Ethernet packets Incoming Ethernet packets per second Number of device interrupts per second Errors per second while sending Ethernet packets Outgoing Ethernet packets per second Number of pages read to physical memory, per second Number of pages written to pagefile(s) and removed from physical memory, per second Number of pages read

Rstatd value ref [2]

avenrun_1 avenrun_5 avenrun_15 colls cswitch cpubusy disk ipkts ierrs intr oerrs opkts pgin pgout

Rstatd type ref [2]

float

sun.mib variable (all counter data types) ref[3]

MISSING

UCD-SNMP-MIB variable ref [4]

ucdavis.laTable.laEntry.laLoad.1 (1m ave) ucdavis.laTable.laEntry.laLoad.2 (5m ave) ucdavis.laTable.laEntry.laLoad.3 (15m ave) MISSING ucdavis.systemStats.ssRawContexts.0 ucdavis.systemStats.ssCpuRawSystem.0 + ucdavis.systemStats.ssCpuRawUser.0 ucdavis.systemStats.ssIORawSent.0 + ucdavis.systemStats.ssIORawReceived.0 if.ifInErrors if.ifInOctets. ucdavis.systemStats.ssRawInterrupts.0 if.ifOutErrors.1 if.ifOutOctets.1 ucdavis.systemStats.ssRawSwapIn.0 ucdavis.systemStats.ssRawSwapOut.0

UCD-SNMP-MIB data type ref [4]

String

Collision rate Context switch rate CPU utilization Disk rate Incoming packets error rate Incoming packets rate Interrupt rate Outgoing packets error rate Outgoing packets rate Page-in rate Page-out rate

counter counter counter counter counter counter counter counter counter counter counter

rsIfCollisions MISSING rsSystemProcessTime + rsUserProcessTime rsDiskXfer1 + rsDskXfer2 + rsDiskXfer3 + rsDiskXfer4 rsIfInErrors rsIfInPackets rsVIntr rsIfOutErrors rsIfOutPackets rsVPagesIn rsVPagesOut

N/A Counter32 (number of switches) Counter32 (100ths of seconds) Counter32 (number of blocks) Counter32 Counter32 Counter32 Counter32 Counter32 Counter32 (number of blocks) Counter32 (number of blocks)

Paging rate

pgin + pgout

counter

rsVPagesIn +

ucdavis.systemStats.ssRawSwapIn.0 +

Counter32

Page 4 of 8

Performance Testing

to physical memory or written to pagefile(s), per second Number of processes being swapped(in)2 Number of processes being swapped(out) Percent of time that the CPU is utilized in system mode Percent of time that the CPU is utilized in user mode

rsVPagesOut Swap-in rate Swap-out rate System mode CPU utilization User mode CPU utilization pswpin pswpout Not explicitly stated Not explicitly stated counter counter Not explicitly stated Not explicitly stated rsVSwapIn rsVSwapOut rsSystemProcessTime rsUserProcessTime

ucdavis.systemStats.ssRawSwapOut.0 MISSING MISSING ucdavis.systemStats.ssCpuRawSystem.0 ucdavis.systemStats.ssCpuRawUser.0

(number of blocks) N/A N/A Counter32 (100ths of seconds) Counter32 (100ths of seconds)

Items shown in white are listed in the SNMP counters QALoad 5.1 collects for Solaris servers. How QALoad should be configured to achieve this is discussed in the next section. Items shown in grey are not listed in the SNMP counters QALoad 5.1 collects for Solaris servers. Customisation to provide these statistics has not yet been addressed. In summary, all the variables required can be collected by at least one of the SNMP agents reviewed. Also, the variables understood by QALoad 5.1 can all be collected via sun.mib (available by default on Solaris 9 and 10).

Though described as number of processes by Mercury, the underlying counters appear to represent number of pages swapped in/out in ref[2] and [3].

Page 5 of 8

Performance Testing

3 Configuring QALoad
QALoad can only collect specific SNMP counters for each type of system under test. The table in section 2 shows, in white, those which can be collected. There is no known way to configure QALoad to collect those shown in grey, or any other custom SNMP counters. Providing the SNMP agent is running on the system under test, and firewall ports are open as need to allow UDP traffic to pass from the QALoad Conductor to the system under test, all the configuration required can be done via the QALoad Conductor GUI. To include the SNMP counters, the Remote Monitoring button is selected on the Machine Configuration tab, then the New button used to launch a dialogue to discover and select the required SNMP counters on selected hosts.

4 Customising QALoad
The table in section 2 shows, in grey, the SNMP values which cannot be collected directly via QALoad. As a result, some indirect method is needed. QALoad allows the user to define custom counters using DEFINE_COUNTER, which may be either instance-type counters (for instantaneous values, or current rates) or cumulative-type counters (for counting errors and events). Values may be written to the QALoad Timing file using VALUE_COUNTER. A script can therefore be written which is run continuously by one virtual user, to collect and store the required values and then sleep for a period. Remaining sections describe approaches which have been considered to producing a QALoad script to collect values for the counters. 4.1 Calling an external program

QALoad allows a command line program (or a remote terminal session3, or remote command execution) to be executed using standard Windows API calls. This is the simpler method as the details of the programming interfaces and protocols are hidden from QALoad. Other monitoring commands could be invoked, instead of, or as well as, SNMP (eg. RPC clients as mentioned in section 2.3).

Following a proof of concept, a QALoad script was implemented to perform the following:
3

Mercury LoadRunner KnowledgeBase article 18748 offers a sample script to run vmstat via a Mercury-specific terminal emulation session. Values for the missing SNMP counters (or equivalent) can all be derived from running vmstat(1M) on the system under test. This would require a telnet port, or similar, to be opened.

Page 6 of 8

Performance Testing

Use _popen() and _pclose() to open a pipe to read SNMP counter values from a host using an SNMP command line program, snmpget, installed on Windows XP as part of NetSNMP. Host names are currently defined in a local datapool. Each virtual user configured on the same Player will monitor a unique host. For those SNMP value which are raw counters, derive a current rate for the SNMP counter value and store in an instance-type custom counter, using a microsecond clock timer to derive rate measurements accurately. For those SNMP values which are already averaged (eg. Load Average), just store in an instance-type custom counter. Sleep for 10 seconds (must be done BEFORE errors are reported, as errors cause DO_Sleep() to return immediately). As UDP is unreliable SNMP packets or responses may not be guaranteed. So record errors as failed transactions and continue. WWW_FATAL_ERROR is called to report the transaction as a fail. The script sets DO_AbortOnError(FALSE) to override any run-time settings.

The script was tested using Conductor and results reviewed using Analyze. The script has been made thread-safe so may be run in process or thread mode. The script has also been tested across a network connection to a Linux server hosting an SNMP agent which implements all the missing counters in section 2. The following tasks remain outstanding: The script may need to be made portable to Linux, and re-tested, if Linux is to be used for Players.

There are a number of dependencies on configuring the test systems and networks: - Net-SNMP would have to be installed on the Player systems and configured with the SNMP community names used by the QALoad script. - SNMP agent would need to be installed on all hosts monitored and SNMP ports open for remote requests from Player systems. - Local datapools are used for information which can be modified within QALoad: SNMP_HOSTNAMES.DAT - Local configuration files which should not be changed during testing must be copied into the datapools directory: SNMP_CONFIG.csv and SNMP_COUNTERS.csv 4.2 Using an SNMP API

WinSNMP is an SNMP programming interface implemented via shared libraries. QALoad also allows external libraries to be included (Options->C++ Compiler Options). This approach has not been explored in detail as others seem more promising. The following API implementations have been identified, but not reviewed in detail: http://www.winsnmp.com - WinSNMP Developers Kit. $199 per runtime copy. Page 7 of 8

Performance Testing

Includes C/C++ code samples. http://www.mg-soft.com MG-SOFT SNMP Software Development Lab $640 (SNMP v3 version) $290 (SNMPv1 and SNMPv2c only) Prices include one developer and 20 runtime licenses Includes C/C++ code samples. Also offers free time-limited evaluation downloads. http://www.agentpp.com - Agent++ V3.5. Source code only. Free SNMP API for C++, unclear if it adheres to WinSNMP.

References
[1] LoadRunner Analysis Users Guide, Mercury [2] Sun Solstice Enterprise Manager 4.1 Answerbook, Sun na.hostperf(8) [3] Solaris 9 Reference Manual man pages section 1(M) - mibiisa(1M) [4] Net-SNMP MIB files (http://www.net-snmp.org) [5] Monitoring CPU usage with OpenSTA via SNMP, http://portal.opensta.org [6] Solaris 10 Sun System Management Agent Developers Guide [7] Solaris 10 Sun System Management Agent Administration Guide

Page 8 of 8