RAC PROACTIVE MONITORING USING OS WATCHER

Alejandro Vargas Principal Support Consultant Oracle Advanced Support Services

OS Watcher Brief Description................................................................................................................................................... 1 Install Steps .............................................................................................................................................................................. 2 First Run ................................................................................................................................................................................... 4 Analyzing The Gathered Information ........................................................................................................................................ 6 End Of The Document ............................................................................................................................................................ 10

OS WATCHER BRIEF DESCRIPTION

OS Watcher is a simple and powerful utility written by Carl Davis, of the Oracle Corporation Center of Expertise. OS Watcher can be used to store a set of Operating System statistics over a period of time in a cyclic way.

These statistics can be used to have a broad vision of the Operating System main performance parameters and are helpful to diagnose potential problems and discover trends that may require proactive intervention. This is a brief description of OS Watcher, the complete guide and scripts can be downloaded from Note 301137.1 on Metalink you will find there a complete description of each statistic with commentaries.

INSTALL STEPS
Download OS watcher from OSW Download The download contains a tar file, untar it on a directory where you want to install the application. OSW store the statistics on text files that . {oracle} /u01/oracle/scripts/av/OSW [rac1] > tar -xvf OSW.tar ./ ./oswnet.sh ./README ./Exampleprivate.net ./tarupfiles.sh ./OSWatcher.sh ./startOSW.sh ./stopOSW.sh ./OSWatcherFM.sh ./OSWg.jar ./topaix.sh ./OSWgREADME ./oswsub.sh

{oracle} /u01/oracle/scripts/av/OSW total 1664 -rwxr-x--x 1 oracle dba 560 Dec -rwxr-x--x 1 oracle dba 1101 Mar -rwxr-x--x 1 oracle dba 409 Mar -rwxr-x--x 1 oracle dba 401 Mar -rw-r----- 1 oracle dba 3029 Mar -rwxr-x--x 1 oracle dba 334 Mar -rwxr-x--x 1 oracle dba 1731 Mar -rwxr-x--x 1 oracle dba 4451 Apr -rwxr-x--x 1 oracle dba 127 Mar -rw-r----- 1 oracle dba 721234 Oct -rw-r----- 1 oracle dba 2768 Oct -rwxr-x--x 1 oracle dba 11784 Oct -rwx------ 1 oracle dba 758784 Jan

[rac1] > ls -ltr 16 2004 18 2005 18 2005 21 2005 22 2005 28 2005 29 2005 5 2005 21 2006 14 2006 14 2006 14 2006 1 11:09 stopOSW.sh startOSW.sh topaix.sh oswsub.sh README oswnet.sh Exampleprivate.net OSWatcherFM.sh tarupfiles.sh OSWg.jar OSWgREADME OSWatcher.sh OSW.tar

The amount of space you will require depends on how long you want the data gathering cycle to be, and how often you will gather information. i.e. you may define a cycle of 7 days and a check to be done every minute. Test the used space after a couple of hours of gathering data with the frequency you choose. OS Watcher came ready to use with a set of preconfigured standard scripts used to gather information, the only exception is private network script. You will need to copy the provided Example private.net and configure it to match you server names: [oracle@rac1 osw]$ cat private.net ######################################################################

# This file contains examples of how to monitor private networks. To # monitor your private networks create an executable file in this same # directory named private.net. Use the example for your host os below. # Make sure not to remove the last line in this file. Your file # private.net MUST contain the rm lock.file line. ###################################################################### #Linux Example ###################################################################### echo "zzz ***"`date` traceroute -r -F -m1 rac1 traceroute -r -F -m1 rac2 rm lock.file # eof private.net

FIRST RUN
When started the first time OS Watcher will check that the OS utilities required to execute the data gathering are available, you may pass two parameters: <seconds between checks> <hours to archive>; if you don't use any parameters then default values of <30> and <48> are used : {oracle} /u01/oracle/scripts/av/OSW [rac1] > ./startOSW.sh 60 7 Testing for discovery of OS Utilities... VMSTAT found on your system. IOSTAT found on your system. MPSTAT found on your system. NETSTAT found on your system.

TOP found on your system. Discovery completed. Starting OSWatcher V2.0.2 on Tue Jun 19 09:10:45 IDT 2007 With SnapshotInterval = 60 With ArchiveInterval = 7 OSWatcher - Written by Carl Davis, Center of Expertise, Oracle Corporation Starting Data Collection... osw heartbeat:Tue Jun 19 09:10:45 IDT 2007 In addition it will create a directory named "archive" containing one subdirectory for each gathered statistic: {oracle} /u01/oracle/scripts/av/OSW/archive [rac1] > ls -ltr total 0 drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswprvtnet drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswvmstat drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswmpstat drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswnetstat drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswiostat drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswtop drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswps drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswmeminfo drwxr-x--x 2 oracle dba 96 Jun 19 09:10 oswslabinfo To stop gathering execute: ./stopOSW.sh

Analyzing the Gathered information

OS Watcher provide a java based graphic interface to visualize the information; you will need to have java 1.4.2 installed to use it. The raw statistics information is stored on text files that span one hour each, that makes easy to drill to the time where a problem was experienced to check if there is any OS related issue. To run the graphic interface you may define an alias on your .cshrc : alias oswg 'setenv PATH $ORACLE_HOME/jre/1.4.2/bin:$PATH; clear; java -jar /u01/oracle/scripts/av/OSW/OSWg.jar -i /u01/oracle/scripts/av/OSW/archive' Executing the alias will bring up a text menu: {oracle} /u01/oracle/scripts/av/OSW/ [rac1] > oswg Starting OSWg V2.0.2 OSWatcher Graph Written by Oracle Center of Expertise Copyright (c) 2006 by Oracle Corporation Parsing Data. Please Wait... Parsing file rac1_iostat_06.19.07.0900.dat ... Parsing file rac1_vmstat_06.19.07.0900.dat ... Parsing Completed. Enter 1 to Display CPU Process Queue Graphs Enter 2 to Display CPU Utilization Graphs Enter 3 to Display CPU Other Graphs

Enter 4 to Display Memory Graphs Enter 5 to Display Disk IO Graphs Enter 6 to Generate All CPU Gif Files Enter 7 to Generate All Memory Gif Files Enter 8 to Generate All Disk Gif Files Enter Enter Enter Enter Enter L T D R Q to to to to to Specify Alternate Location of Gif Directory Specify Different Time Scale Return to Default Time Scale Remove Currently Displayed Graphs Quit Program

Please Select an Option:

When selecting IO graphs you will get a list of devices with their service time, you can choose the one you want to check: Please Select an Option:5 The Following Devices and Average Service Times Are Ready to Display: Device Name sdo Average Service Times in Milliseconds 4.981923076923077

sdm sdu sdz sdg sde sdx sdh sdt sdy sdk sdr sdl sdv sdw sdi sdab sdj sdd sdq sda sdf sds sdaa sdb sdc sdn sdp

4.726923076923077 4.4326923076923075 4.331153846153845 4.323846153846154 4.319230769230769 4.262692307692308 4.209999999999999 4.205769230769231 4.073076923076924 3.9811538461538465 3.9619230769230778 3.9057692307692307 3.8734615384615387 3.8146153846153843 3.794615384615385 3.7780769230769233 3.743461538461539 3.147692307692308 2.8203846153846155 2.7315384615384612 0.6538461538461539 0.3076923076923077 0.0 0.0 0.0 0.0 0.0

End of the Document