Workload Control Center Performance Tuning

This document includes recommendations and best practices for performance tuning for CA Workload Control Center (WCC), for the following releases: CA Workload Control Center r11.1 CA Workload control Center r11.1 SP1 CA Workload control Center r11.1 SP2

The document includes general recommendations, feature-specific recommendations, and release-specific information, as well as procedures and checklists for the recommendations.

General Recommendations
General recommendations for performance tuning are as follows: 1. Make sure that the WCC server machine meets the recommended specifications identified in the README. The WCC readme includes details of hardware requirements in the System Requirements section. 2. Remove unused items from the Quick Start menu using EEM security. 3. Modify the default maxThreads and maxmemory settings for each Tomcat server. The maxThreads parameter on Windows is located in %CA_WCC_INSTALL_LOCATION%\*Server\conf\server.xml or on *nix it is located in $CA_WCC_INSTALL_LOCATION/*Server/conf/server.xml. This parameter should be set to 2x the maximum expected number of concurrent users for the feature. 4. The maxmemory parameter should be set to a maximum of 900 MB. You must ensure that there is adequate memory on the machine for the settings; the JVM will not load if maxmemory cannot be allocated. This parameter is located in %CA_WCC_INSTALL_LOCATION%\*Server\conf\wrapper.conf. For example, C:\Program Files\CA\Workload Control Center\MonitorServer\conf\wrapper.conf. Job Status Console has a slightly different location: %CA_WCC_INSTALL_LOCATION%\JobStatusConsoleServer\JobSServer\conf\wra pper.conf For *nix the path is $CA_WCC_INSTALL_LOCATION/*Server/conf/wrapper.conf. For example, /opt/CA/WorkloadCC/MonitorServer/conf/wrapper.conf.

Page 1 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

Job Status Console has a slightly different location: $CA_WCC_INSTALL_LOCATION/JobStatusConsoleServer/JobSServer/conf/wrappe r.conf Notes: Windows 2003 Server edition does not support more than 4 GB RAM. To utilize more than 4 GB of RAM, you must use Windows 2003 Enterprise. If the values for maxmemory and initmemory arent specified, the default for the jvm is used. 2 for init, 84 for max. 5. Reduce the amount of storage used for logging. The same wrapper.conf files in the previous item also contain a wrapper.logfile.maxfiles=0 statement. This setting controls the number of rolled logs. The log is rolled based on size and 0 indicates no limit on the number of logs. Change the setting to control how many files will be kept for this log. 6. Limit logging to Error/Info levels Workload Control Center includes extensive logging for all components. By default, logging levels are set to INFO or ERROR. To optimize performance, ensure that you do not set logging values to DEBUG. These logs are located in %CA_WCC_INSTALL_LOCATION%\Logs\Application (Windows) or $CA_WCC_INSTALL_LOCATION /Logs/Application (*NIX) and there is a log4j.properties file in each subfolder that controls these logging settings. We recommend that if you change any logging settings, you first make a backup copy of the original log4j.properties file. When you have made changes to the log4j.properties file, save the file then stop and start the associated service(s). 7. Check the network as well as machine usage, especially if a subset of users have problems Occasionally, there will be a problem with network bandwidth in an area of the enterprise. If there is a subset of users who are having problems, a simple test is to perform a ping to see what the transit time is between the users machine and the WCC server. 8. Resolve resource contention issues for virtual servers and shared servers. If WCC is installed on a shared server or virtual server, ensure that there are no resource contention problems. For example, if WCC is installed on a virtual server that hosts multiple environments and you experience periodic performance drops, verify that the other environments are not causing resource utilization spikes. 9. Ensure there are no database connection limitations WCC utilizes database connections for AutoSys 4.x. For AutoSys 4.x, ensure that any limits to the number of database connections are not being violated. For CA-7, you must ensure that ACBs are sized appropriately. We recommend that you work with CA Support to set this up. An approximate number is:

Page 2 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

Number of concurrent users + connection pool + CCI terminals for batch processing 10. Modify the initial and maximum heap sizes on client computers (Job Flow Monitoring only) 11. Ensure that there are no performance bottlenecks with EEM, due to Active Directory connectivity or EEM event or audit logging. 12. On *NIX operating systems, set ULIMIT n = 1024.

Feature-specific Recommendations
Each feature contains an overview of how the cache is constructed, plus specific recommendations for optimizing performance for that feature.

Job Editor
Job Editor Cache
The initial expansion of server/instance fetches object names. This data goes into user-based cache on the WCC server. o There are some properties cached on the WCC server, but these are minimal (group, application) Selecting an object fetches the properties of that object. The property data goes into the client-based cache. Saving an object refreshes the properties of the object in the cache. Executing a Refresh on the server will refresh the object names on the users server-based cache and will flush the property data from the cache.

Job Editor Recommendations

Follow the General recommendations For CA 7, do the following: o Set the Connection Pool Size to an appropriate size, based on the number of expected concurrent users who will be using the connection pool. o Ensure that the number of ACBs on the mainframe is set to an appropriate value

Quick Edit and Application Editor

Quick Edit and Application Editor are only available in WCC r11.1 Sp1 and WCC r11.1 SP2, for AutoSys r11 servers. For both Quick Edit and Application Editor, you can specify whether to enable an AutoSys API performance-enhanced search or the original ASAPI search. In the README, please review Section 5.6 Application Editor and Quick Edit User Interface Considerations for details on how to enable the performance-enhanced search. You must also have at least AutoSys r11 SP2 Incremental 2 to use this.

Page 3 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

Job Flow Monitoring
Job Flow Monitoring Cache
Job flows are processed on the server. Each job flows data is maintained separately. All users share the same data for a job flow that is on production. User ID: o CA-7 and AutoSys r11: the Job Flow is activated under the monitor ID (CA 7 and AutoSys r11). o AutoSys 4.x: the Job Flow is activated under the database user ID. Status data and structure updates (dependency changes) are fetched under the user ID, and are updated across all users. For the following actions there is additional load because the users credentials are utilized, rather than the common monitoring ID: o Executing job actions (Start, Force Start, Requeue, Cancel, etc.) o Accessing the Requirements View o Accessing the Properties View

Job Flow Monitoring Recommendations

Follow the General recommendations Modify the following parameters, as follows: o Increase Number of Threads for WCC r11.1 and WCC r11.1 SP1. The number of threads should be greater than or equal to the number of Job Flows on production. o For WCC r11.1 SP2 only, decrease Number of Threads. The recommended value for a typical enterprise is 5. o For WCC r11.1 SP2 only, set Thread Pool Size to a low value. The recommended value for a typical enterprise is 10. o Increase Number of Caches based on number of Concurrent users. o Reduce Session Idle Timeout o Lengthen Update Interval values (AutoSys and CA 7) o CA-7 Increase Connection Pool Size Flow Rendering only: o Reduce the Job Flow Maximum Job Number, Job Flow Maximum Number of Levels, Job Flow Maximum Cutoff Level o Reduce the Monitoring Refresh Interval to provide a faster refresh rate in the Job Flow Monitoring applet. Modify the initial and maximum heap sizes on client computers to improve flow rendering.

Job Status Console

Job Status Console Cache
The Job Status Console cache is user ID-based. If data is available in the cache, the 2nd user accessing the data will get the cached data rather than a new fetch. Cache is automatically refreshed when a user hits Refresh in a view. More complex/more options than JFM or Job Editor o Configuration/Environment/Job Status Console Cache Properties Properties

Page 4 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

o Caching for AutoSys - Enable Cache Size o Set to # views + expected # of concurrent users of JSC o No downside to setting it higher rather than lower. Cache Validity Period o set higher to use cache more. Low validity period = very fresh data, but will cost more in memory usage. o Recommended is the refresh rate on your views. Cache Timeout o Set to validity period. Maintenance Interval runs check to remove. Limit any Alert or Alarm filters to jobs in the view.

Job Status Console Recommendations

Follow the general recommendations, including optimizing the maxThreads and maxmemory settings. The following optimizations need to be made to the JVM settings for WCC. These need to be made in %CA_WCC_INSTALL_LOCATION%\JobStatusConsoleServer\JobSServer\conf\ wrapper.conf on Windows and $CA_WCC_INSTALL_LOCATION/JobStatusConsoleServer/JobSServer/conf/wra pper.conf on *nix. wrapper.java.initmemory=900 wrapper.java.maxmemory=900 set.AS_THREAD_STACK=65536 You will see the wrapper.java.initmemory and wrapper.java.maxmemory lines in the wrapper.conf file. Modify the values to the values in this update, and then add set.AS_THREAD_STACK=65536 immediately below. Save the wrapper.conf file, and restart the two Job Status Console services. On *NIX, ULIMIT -n should be set to 1024 The Global Session is for the Views portlet. Utilizing this makes the Views portlet much more efficient. You must provide a valid Global User ID/password if you enable the Global Session. Alerts DB o For AutoSys only enterprises, alerts may not be required o Decrease Alert Deletion Threshold value. o Set Alert Deletion Threshold to 0 and restart JSC to clean up all the existing alerts View Size Calculation o Cost per job, cost per user ID. To accommodate more views, decrease the Session Inactivity Timeout value. Consider increasing the Portlet Refresh Interval and update interval for views on Development/Test instances. To accommodate more concurrent users, increase the following parameters: o Pool Size: set to Connection Pool Size o Connection Pool Size: set at 5 to 10. o Model Worker Pool Size controls threading. Increase this value above the default (5) if the WCC server is NOT CPU-bound.

Page 5 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

EEM (CA Embedded Entitlements Manager)
Ensure that the CAELM application is installed and operational Review event policies in both the AutoSys and WCC applications to ensure that these are correctly formulated. If you are using Active Directory as your EEM user repository, ensure that you set appropriate filters in the EEM/AD connection to limit the allowed users to groups and individuals that actually use WCC, rather than retrieving the entire AD.

Release-specific Recommendations
Workload Control Center r11.1
Modify EEM logging settings, as described in the Procedures section. Apply WCC r11.1 patch for JSC opening too many files when processing views.

Workload Control Center r11.1 SP1

Apply WCC r11.1 patch for JSC opening too many files when processing views.

Scheduling Engine Recommendations

AutoSys 4.5.x
If you are experiencing problems with the number of concurrent users, please check with AutoSys Support. There is a known issue with RCS and concurrent users for which there is a fix.

AutoSys r11
AutoSys r11 utilizes an application server to manage secure communication with the AutoSys database. The application server is typically installed on the AutoSys Scheduler machine. If the machine where the application server is installed is heavily used, the machine may become CPU-bound. To maximize performance, install the instance of the application server that is used by WCC to a lower-load machine. There is a known issue in the AutoSys API that affects both Job Editor and Job Flow Monitoring. Please check with AutoSys support for this fix. We recommend that you apply the most recent maintenance (Service Packs, incremental) available for AutoSys. Please ensure that in addition to applying the maintenance to AutoSys itself, you also apply the AutoSys client libraries to the WCC server, as well as any common component maintenance

Page 6 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

(including SSA updates) available in the AutoSys maintenance package, according to the instructions in the README file for the package. We recommend you apply the AutoSys API Memory Leak and FDS patch on WCC Server.

CA 7 (all releases)
Ensure that the number of ACBs on the mainframe is set to an appropriate value. Ensure that CPS parameters are set appropriately. Typical values for these parameters are:

o o
o

SET SLIMIT=50 -- Maximum Active Sessions SET SESSTOUT=60 -- Session Timeout (in minutes) no activity SET PASSMODE=DORM -- CPS Pass Ticket Processing (PASS, DORM, FAIL)

Procedures
The following section details procedures referenced above.

Increase the initial and maximum heap sizes on a client computer

If you are experiencing problems with loading very large flows in Job Flow Monitoring, you can increase the initial and maximum heap size in the Java browser plug-in. For JRE 1.5.x and before: 1. Click Start, Settings, Control Panel. The Control Panel window appears. 2. Double click the Java Plug-in icon. The Java Plug-in Control Panel window appears. 3. Click the Advanced tab of the Java Plug-in Control Panel. 4. In the Java Runtime Parameters field, enter the following: -Xms256m -Xmx400m 5. Click Apply. The initial and maximum heap sizes are updated. For JRE 1.6+ 1. Click Start, Settings, Control Panel. The Control Panel window appears. 2. Double click the Java icon. The Java Control Panel window appears. 3. Click the Java tab, and then click the View button in the Java Applet Runtime Settings panel. The Java Runtime Settings dialog opens. 4. For the most recent version of the JRE (plus any other versions that are installed on the machine), enter the following in the Java Runtime Parameters column: 5. -Xms256m -Xmx400m

Page 7 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

6. Click OK. The Java Runtime Settings dialog is dismissed. 7. Click Apply then click OK to close the Java Control Panel window. Note: The Xmx parameter can cause problems since the Java Virtual Machine will fail to load if it cant get enough contiguous memory from the OS to satisfy the amount requested in the Xmx param. This is typically a problem on Windows when the Xmx is set to 512m or higher.

Enabling the Quick Edit and Application Editor advanced search feature
To enable the enhanced AutoSys API search feature, you must set the associated parameter value in the EditorConfiguration.properties file. This file is located as follows:
Application Editor (Windows) %CA_WCC_INSTALL_LOCATION%\AdminServer\webapps\AppEditor\WEBINF\classes\com\ca\wcc\editor\resources

Quick Edit (Windows)

%CA_WCC_INSTALL_LOCATION%\UIFrameworkServer\webapps\QuickEdit\WEB
-INF\classes\com\ca\wcc\editor\resources

Application Editor (UNIX)

$CA_WCC_INSTALL_LOCATION/AdminServer/webapps/AppEditor/WEBINF/classes/com/ca/wcc/editor/resources

Quick Edit (UNIX)

$CA_WCC_INSTALL_LOCATION/UIFrameworkServer/webapps/QuickEdit/WEBINF/classes/com/ca/wcc/editor/resources

To enable the AutoSys API (ASAPI) enhanced search, locate the following statements: # Use ASAPI enhanced search # 'Yes' to use ASAPI performance enhancement # 'No' to use original ASAPI search editor.enhanced.search=No As indicated, change the editor.enhanced.search to Yes, then save the file and restart the appropriate service (CA Workload Control Center UI Framework Server and/or CA Workload Control Center Scheduling Server).

Page 8 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

Security Component (EEM) Logging
This procedure will prevent the eTrust security component eTrust EEM from logging summary messages 1. Modify <eTrust Directory Home>\dxserver\config\logging\iTechPoz<hostname>.dxc on Windows or <eTrust Directory Home>/dxserver/config/logging/iTechPoz-<hostname>.dxc on *NIX. 2. Comment out "set summary-log = "logs/$s.log";" by placing a # at the start of the line 3. Modify <eTrust Directory Home>\dxserver\config\logging\iTechPoz<hostname>-Router.dxc on Windows or <eTrust Directory Home>/dxserver/config/logging/iTechPoz-<hostname>-Router.dxc on *NIX. 4. Comment out "set summary-log = "logs/$s.log";" by placing a # at the start of the line

Security Component (EEM) Updates

WCC r11.1 and r11.1 SP1 are delivered with the Common Components DVD, which contains eTrust Embedded Entitlements Manager Server 8.3. We recommend that you update the EEM Server release to EEM Server 8.3 SR01 or above, preferably to EEM 8.4 SR02. EEM Server 8.3 SR01 update is available from the AutoSys r11 SP1 Incremental 1 patch. There are instructions in the AutoSys r11 SP1 Incremental 1 for the EEM Server 8.3 SR01 upgrade. EEM Server 8.4 SR01 update is available from the AutoSys r11 SP2 Incremental 2 patch. There are instructions in the AutoSys r11 SP2 Incremental 2 for the EEM Server 8.4 SR02 upgrade. EEM Server 8.4 SR02 update is available from the AutoSys r11 SP3 media. In addition to the EEM Server upgrade, we recommend that you update the EEM safe.jar that is delivered with WCC to the safe.jar from EEM SDK 8.3 SR01 or above. Check support.ca.com for the latest patches or contact CA Technical Customer Support. Important! Refer to CA WCC Certification Information on the latest supported versions of EEM Server for various WCC versions.

Linux Kernel
On Linux operating systems only, the Linux kernel used by default is 2.4. We recommend that you update this to use the 2.6 kernel, as there are significant efficiencies in the threading model embedded in the 2.6 kernel. Modify /etc/init.d/igatewayd and IGW_LOC/S99igateway as follows: 1. Stop iGateway. 2. Open /etc/init.d/igatewayd, comment out the line LD_ASSUME_KERNEL=2.4.1, and save. 3. Open $IGW_LOC/S99igateway, comment out the line LD_ASSUME_KERNEL=2.4.1, and save. 4. Restart iGateway.

Page 9 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

Set the JSC Global Session Parameters
1. Open the WCC Configuration Manager. 2. Click the Environment tab, and then click Job Status Console Server. The Job Status Console Server page opens. 3. From the Use Global Session drop-down list, choose True. 4. In the Global User ID field, enter the ID for the Job Status Console global user. 5. In the Global User Password field, enter the password for the Job Status Console global user. 6. In the Global Confirm User Password field, enter the password for the Job Status Console global user. 7. Click Save. 8. Click the Deploy tab. The Deploy page opens with the Deploy link selected by default. 9. In the Deploy Comments field, type appropriate text to indicate your changes. 10. Click Deploy. The comment you typed now appears in the Last Deploy Comments field. 11. Click the Services link. The Services page opens. This page will indicate what services need to be restarted following the deploy. Restart the indicated services. Important! If you do not plan to use the Global Session, make sure that you change the Use Global Session parameter to False. By default, the value for this parameter is True. Guidelines for using the JSC Global User: The breakeven point is 2 watchers for a view. Watchers are users who just watch the views portlet and dont actively use a view. Active view users are irrelevant because they work the same in both methods. Remember also that if you have access to 5 views but actively work in 1, you are a watcher of 4 views. If you have 800 views and each belongs to 1 unique user and no one else, you are better off disabling the global user. If you have 800 views and 3 watchers for each view, you are better off using the global user. In between those two scenarios, you need to know exact usage patterns to know which method makes the least calls to AutoSys and uses the least amount of memory. Note: The CA 7 user must have list and FOJOB access. The AutoSys user can have read-only access.

Checklists
General Recommendations
# 1 2 3 4 5 Action Make sure that the WCC server machine meets the recommended specifications identified in the README. Remove unused features from Quick Start using EEM policies Remove unused portlets from the workplace Modify the default maxThreads and maxmemory settings for each Tomcat server Limit logging to Error/Info levels

Page 10 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

6 7 8 9 Check the network as well as machine usage, especially if a subset of users have problems Resolve resource contention issues for virtual servers and shared servers. Ensure there are no database connection limitations On *NIX, set ULIMIT n = 1024

Security Component (CA EEM)

# 1 2 3 4 5 6 Action Ensure that logging does not include summary messages Update to the latest available EEM version and update the WCC safe.jar as directed by Support. If you are using WCC r11.1 SP2, this will include the EEM 8.4 SR02 safe.jar. On Linux operating systems, modify the Linux kernel parameters to ensure that the 2.6 threading model is used. If you are using Active Directory as the EEM repository, ensure that you have set appropriate filters to limit the scope of the Active Directory return. Verify that the CAELM application is installed and running on your EEM instance Review EEM audit logging/logging policies to ensure that these are appropriate.

Job Editor
# 1 2 3 4 5 6 Action Modify the default maxThreads settings for the Job Editor Tomcat instance (AdminServer) Modify the default maxmemory settings for the Job Editor Tomcat instance (AdminServer) Ensure there are no database connection limitations Limit logging to Error/Info levels CA 7 Only: Set the Connection Pool Size to an appropriate size, based on the number of expected concurrent users who will be using the connection pool CA 7 Only: Ensure that the number of ACBs on the mainframe is set to an appropriate value

Quick Edit and Application Editor

# 1 Action Enable the performance enhanced search in WCC and AutoSys.

Page 11 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

Job Flow Monitoring
Base Job Flow Monitoring
# 1 2 3 4 5 6 7 Action Modify the default maxThreads settings for the Job Flow Monitoring Tomcat instance (MonitorServer) Modify the default maxmemory setting for the Job Flow Monitoring Tomcat instance (MonitorServer) to 900. Monitoring Parameters: Increase Number of Threads Monitoring Parameters: Decrease Number of Caches Monitoring Parameters: Reduce Session Idle Timeout Monitoring Parameters: Lengthen Update Interval values (AutoSys and CA 7) Monitoring Parameters, CA 7 Only: Set the Connection Pool Size to an appropriate size, based on the number of expected concurrent users who will be using the connection pool

Job Flow Monitoring - Flow Rendering Only

# 1 2 3 4 Action Monitoring Parameters: Reduce the Number Monitoring Parameters: Reduce the Levels Monitoring Parameters: Reduce the Level Client Computers: Modify the initial the client Java browser plug-in Job Flow Maximum Job Job Flow Maximum Number of Job Flow Maximum Cutoff and maximum heap sizes for

Job Status Console

# 1 2 3 4 5 6 7 8 9 Action Modify the default maxThreads settings for the Job Status Console Tomcat instance (JobStatusConsoleServer) Modify the default maxmemory setting for the Job Status Console Tomcat instance (JobStatusConsoleServer) to 900. Enable the Global Session and Global User Alerts Database: For AutoSys only enterprises, determine if alerts are needed, or if alarms meet the needs of the enterprise Alerts Database: Decrease Alert Deletion Threshold value Alerts Database: Set Alert Deletion Threshold to 0 and restart JSC to clean up all the existing alerts View optimization: Decrease the Session Inactivity Timeout View optimization: Consider increasing the Portlet Refresh Interval and Update interval for views on Development/Test instances Concurrent user optimization: Set Pool Size to the value of the Connection Pool Size

Page 12 of 13

Copyright 2009 CA. All rights reserved.

Workload Control Center Performance Tuning

10 11 Concurrent user optimization: Set Connection Pool Size to 5 to 10 Concurrent user optimization: Model Worker Pool Size (controls threading) - increase this value above the default (5) if the WCC server is NOT CPU-bound

CA 7 (all releases)
# 1 Action Set ACBs to an appropriate size

AutoSys 4.5.x
# 1 Action Apply patch for RCS concurrent users

AutoSys r11
# 1 2 Action Ensure that the machine on which the AutoSys Application Server instance utilized by UWCC is installed is not CPU-bound. If it is, consider relocating this Application Server instance. Apply the latest available AutoSys maintenance package. This must be applied to the WCC server as well as to the AutoSys server.

Page 13 of 13

Copyright 2009 CA. All rights reserved.