debugging cluster failure - network issue

When preparing for a nice test scenario I could not help noticing a few needed services where down.
In my cluster I have the rdbms ORACLE_HOME's located on acfs and the acts filesystems where
down. Not only that, most of the cluster services where down. So where to start looking? The cluster
is a small 2 node cluster using iscsi disks and a regular ethernet network for all connections, running
oel5.7 and Oracle 11.2.0.3.4.

First thing to check is do I have the storage?

?
1
2 [root@vol57-1 ~]# ocrcheck
3 Status of Oracle Cluster Registry is as follows :
4 Version : 3
5 Total space (kbytes) : 262120
Used space (kbytes) : 3516
6 Available space (kbytes) : 258604
7 ID : 220286358
8 Device/File Name : +DATA
9 Device/File integrity check succeeded
10
Device/File not configured
11
12 Device/File not configured
13
14 Device/File not configured
15
16 Device/File not configured
17
18 Cluster registry integrity check succeeded
19
20 Logical corruption check succeeded
21

So, the storage is not the problem, I have access and it looks to be in a decent state, looking from
node vol57-1. How about the other node, vol57-2?

?
1
2 [root@vol57-2 ~]# ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
3 PROC-26: Error while accessing the physical storage
4 ORA-29701: unable to connect to Cluster Synchronization Service
5

node 2 has more problems.

How are the cluster services doing?

?
1 [root@vol57-1 ~]# crsctl check crs
2 CRS-4638: Oracle High Availability Services is online
3 CRS-4535: Cannot communicate with Cluster Ready Services
4 CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager
5

Normally we want it all to be online. How is node 2?

?
1 [root@vol57-2 ~]# crsctl check crs
2 CRS-4638: Oracle High Availability Services is online
3 CRS-4535: Cannot communicate with Cluster Ready Services
4 CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
5

Can we start the cluster services? If not, they could give error messages that might be helpful.

?
1
2 [root@vol57-1 ~]# crsctl start cluster
3 CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vol57-1'
CRS-2672: Attempting to start 'ora.crsd' on 'vol57-1'
4 CRS-2676: Start of 'ora.crsd' on 'vol57-1' succeeded
5 CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the fo
6 Start action for HAIP aborted. For details refer to "(:CLSN00107:)" in "/data/oracle/gr
7 1/agent/ohasd/orarootagent_root/orarootagent_root.log".
CRS-2674: Start of 'ora.cluster_interconnect.haip' on 'vol57-1' failed
8 CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'vol57-1'
9 CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'vol57-1' succeeded
10 CRS-4000: Command Start failed, or completed with errors.
11

hmm, that does not make me very happy but there is a nice pointer. The interconnect looks to be in
trouble. Let's see what it in the orarootagent_root.log:

?
1 2013-01-04 13:44:43.669: [ AGFW][1111320896] {0:0:113} Agent received the message: RE
2013-01-04 13:44:43.670: [ AGFW][1111320896] {0:0:113} Preparing START command for: o
2 2013-01-04 13:44:43.670: [ AGFW][1111320896] {0:0:113} ora.cluster_interconnect.haip
3 2013-01-04 13:44:43.671: [ora.cluster_interconnect.haip][1104529728] {0:0:113} [start] (
4 2013-01-04 13:44:43.671: [ora.cluster_interconnect.haip][1104529728] {0:0:113} [start] N
5 2013-01-04 13:44:43.671: [ AGFW][1111320896] {0:0:113} Agent received the message: RE
6 2013-01-04 13:44:43.671: [ AGFW][1111320896] {0:0:113} Preparing START command for: o
2013-01-04 13:44:43.671: [ AGFW][1111320896] {0:0:113} ora.crsd 1 1 state changed fro
7 2013-01-04 13:44:43.671: [ora.cluster_interconnect.haip][1104529728] {0:0:113} [start] N
8 2013-01-04 13:44:43.671: [ USRTHRD][1104529728] {0:0:113} Thread:[NetHAMain]start {
9 2013-01-04 13:44:43.671: [ USRTHRD][1104529728] {0:0:113} Thread:[NetHAMain]start }
10 2013-01-04 13:44:43.672: [ USRTHRD][1109219648] {0:0:113} [NetHAMain] thread started
2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] (:CLSN00107:) clsn_age
11 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Arg Value = reboot
12 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] getOracleHomeAttrib: o
13 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Utils::getCrsHome crsH
14 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Utils::execCmd 1 USR_O
15 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Utils::getCrsHome crsH
2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Adding Environment Var
16 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
17 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
18 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Adding Environment Var
19 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
20 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Adding Environment Var
21 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Adding Environment Var
22 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] Utils:execCmd action =
23 2013-01-04 13:44:43.673: [ora.crsd][1123928384] {0:0:113} [start] getOracleHomeAttrib: o
24 2013-01-04 13:44:43.678: [ora.crsd][1123928384] {0:0:113} [start] execCmd ret = 0
2013-01-04 13:44:43.679: [ USRTHRD][1123928384] {0:0:113} Thread:[DaemonCheck:crsd]stop
25 2013-01-04 13:44:43.679: [ USRTHRD][1123928384] {0:0:113} Thread:[DaemonCheck:crsd]stop
26 2013-01-04 13:44:43.680: [ USRTHRD][1109219648] {0:0:113} Ocr Context init default level
27 2013-01-04 13:44:43.680: [ default][1109219648]clsvactversion:4: Retrieving Active Versi
28 2013-01-04 13:44:43.686: [ USRTHRD][1109219648] {0:0:113} HAIP: mbr num is 0.
[ CLWAL][1109219648]clsw_Initialize: OLR initlevel [70000]
29 2013-01-04 13:44:43.722: [ USRTHRD][1109219648] {0:0:113} HAIP: initializing to 1 interf
30 2013-01-04 13:44:43.722: [ USRTHRD][1109219648] {0:0:113} HAIP: configured to use 1 inte
31 2013-01-04 13:44:43.725: [ USRTHRD][1109219648] {0:0:113} use all detected INF
32 2013-01-04 13:44:44.681: [ora.crsd][1123928384] {0:0:113} [start] without returnbuf
33 2013-01-04 13:44:44.683: [ora.crsd][1123928384] {0:0:113} [start] clsdmc_respget return:
2013-01-04 13:44:44.683: [ora.crsd][1123928384] {0:0:113} [start] getOracleHomeAttrib: o
34 2013-01-04 13:44:44.684: [ora.crsd][1123928384] {0:0:113} [start] PID 19165 from /data/o
35 2013-01-04 13:44:44.684: [ora.crsd][1123928384] {0:0:113} [start] }DaemonAgent::start
36 2013-01-04 13:44:44.684: [ora.crsd][1123928384] {0:0:113} [start] (:CLSN00107:) clsn_age
37 2013-01-04 13:44:44.684: [ AGFW][1123928384] {0:0:113} Command: start for resource: o
38 2013-01-04 13:44:44.685: [ AGFW][1111320896] {0:0:113} Agent sending reply for: RESOU
2013-01-04 13:44:44.685: [CLSFRAME][1782259808] TM [MultiThread] is changing desired thr
39 2013-01-04 13:44:44.687: [ora.crsd][1123928384] {0:0:113} [check] clsdmc_respget return:
40 2013-01-04 13:44:44.688: [ USRTHRD][1123928384] {0:0:113} Thread:[DaemonCheck:crsd]start
41 2013-01-04 13:44:44.688: [ USRTHRD][1123928384] {0:0:113} Thread:[DaemonCheck:crsd]start
42 2013-01-04 13:44:44.688: [ora.crsd][1123928384] {0:0:113} [check] DaemonAgent::check ret
2013-01-04 13:44:44.688: [ AGFW][1111320896] {0:0:113} ora.crsd 1 1 state changed fro
43 2013-01-04 13:44:44.688: [ AGFW][1111320896] {0:0:113} Started implicit monitor for [
44 2013-01-04 13:44:44.688: [ AGFW][1111320896] {0:0:113} Agent sending last reply for:
45 2013-01-04 13:44:45.690: [ USRTHRD][1119725888] {0:0:113} Thread:[DaemonCheck:crsd]Daemo
46 2013-01-04 13:44:45.690: [ USRTHRD][1119725888] {0:0:113} Thread:[DaemonCheck:crsd]Initi
47 2013-01-04 13:44:45.690: [ USRTHRD][1119725888] {0:0:113} Check action requested by agen
2013-01-04 13:44:45.690: [ USRTHRD][1119725888] {0:0:113} Thread:[DaemonCheck:crsd]isRun
48 2013-01-04 13:44:45.690: [ AGFW][1111320896] {0:0:113} Agent received the message: RE
49 2013-01-04 13:44:45.690: [ AGFW][1111320896] {0:0:113} Preparing CHECK command for: o
50 2013-01-04 13:44:45.691: [ COMMCRS][1113422144]clscsendx: (0x12b9a890) Physical connecti
51
52 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
2013-01-04 13:44:45.691: [ora.crsd][1113422144] {0:0:113} [check] Error = error 11 encou
53 2013-01-04 13:44:45.691: [ora.crsd][1113422144] {0:0:113} [check] Calling PID check for
54 2013-01-04 13:44:45.691: [ora.crsd][1113422144] {0:0:113} [check] Process id 19165 trans
55 2013-01-04 13:44:45.692: [ COMMCRS][1113422144]clscsendx: (0x12b9a890) Connection not ac
56
57 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
58 2013-01-04 13:44:45.692: [ora.crsd][1113422144] {0:0:113} [check] Error = error 6 encoun
2013-01-04 13:44:45.692: [ora.crsd][1113422144] {0:0:113} [check] DaemonAgent::check ret
59 2013-01-04 13:44:45.692: [ AGFW][1111320896] {0:0:113} ora.crsd 1 1 state changed fro
60 2013-01-04 13:44:45.692: [ AGFW][1111320896] {0:6:90} Generating new Tint for unplann
61 2013-01-04 13:44:45.692: [ AGFW][1111320896] {0:6:90} Agent sending message to PE: RE
2013-01-04 13:44:45.700: [ AGFW][1111320896] {0:6:90} Agent received the message: RES
62 2013-01-04 13:44:45.700: [ AGFW][1111320896] {0:6:90} Preparing START command for: or
63 2013-01-04 13:44:45.700: [ AGFW][1111320896] {0:6:90} ora.crsd 1 1 state changed from
64 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] (:CLSN00107:) clsn_agen
65 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Arg Value = reboot
2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] getOracleHomeAttrib: or
66 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Utils::getCrsHome crsHo
67 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Utils::execCmd 1 USR_OR
68 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Utils::getCrsHome crsHo
69 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Adding Environment Vari
70 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
71 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Adding Environment Vari
72 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
73 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
74 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Adding Environment Vari
2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Adding Environment Vari
75 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] Utils:execCmd action =
76 2013-01-04 13:44:45.701: [ora.crsd][1113422144] {0:6:90} [start] getOracleHomeAttrib: or
77 2013-01-04 13:44:45.705: [ora.crsd][1113422144] {0:6:90} [start] execCmd ret = 0
78 2013-01-04 13:44:45.705: [ USRTHRD][1113422144] {0:6:90} Thread:[DaemonCheck:crsd]stop {
79 2013-01-04 13:44:45.705: [ USRTHRD][1113422144] {0:6:90} Thread:[DaemonCheck:crsd]stop }
2013-01-04 13:44:45.729: [ USRTHRD][1109219648] {0:0:113} use all detected INF
80 2013-01-04 13:44:46.709: [ora.crsd][1113422144] {0:6:90} [start] without returnbuf
81 2013-01-04 13:44:46.711: [ora.crsd][1113422144] {0:6:90} [start] clsdmc_respget return:
82 2013-01-04 13:44:46.711: [ora.crsd][1113422144] {0:6:90} [start] getOracleHomeAttrib: or
83 2013-01-04 13:44:46.711: [ora.crsd][1113422144] {0:6:90} [start] PID 19179 from /data/or
84 2013-01-04 13:44:46.711: [ora.crsd][1113422144] {0:6:90} [start] }DaemonAgent::start
2013-01-04 13:44:46.711: [ora.crsd][1113422144] {0:6:90} [start] (:CLSN00107:) clsn_agen
85 2013-01-04 13:44:46.711: [ AGFW][1113422144] {0:6:90} Command: start for resource: or
86 2013-01-04 13:44:46.712: [ AGFW][1111320896] {0:6:90} Agent sending reply for: RESOUR
87 2013-01-04 13:44:46.712: [ora.crsd][1113422144] {0:6:90} [check] clsdmc_respget return:
88 2013-01-04 13:44:46.714: [ USRTHRD][1113422144] {0:6:90} Thread:[DaemonCheck:crsd]start
2013-01-04 13:44:46.714: [ USRTHRD][1113422144] {0:6:90} Thread:[DaemonCheck:crsd]start
89 2013-01-04 13:44:46.714: [ora.crsd][1113422144] {0:6:90} [check] DaemonAgent::check retu
90 2013-01-04 13:44:46.714: [ AGFW][1111320896] {0:6:90} ora.crsd 1 1 state changed from
91 2013-01-04 13:44:46.715: [ AGFW][1111320896] {0:6:90} Started implicit monitor for [o
92 2013-01-04 13:44:46.715: [ AGFW][1111320896] {0:6:90} Agent sending last reply for: R
93 2013-01-04 13:44:47.716: [ USRTHRD][1107118400] {0:6:90} Thread:[DaemonCheck:crsd]Daemon
2013-01-04 13:44:47.716: [ USRTHRD][1107118400] {0:6:90} Thread:[DaemonCheck:crsd]Initia
94 2013-01-04 13:44:47.716: [ USRTHRD][1107118400] {0:6:90} Check action requested by agent
95 2013-01-04 13:44:47.716: [ USRTHRD][1107118400] {0:6:90} Thread:[DaemonCheck:crsd]isRunn
96 2013-01-04 13:44:47.716: [ AGFW][1111320896] {0:6:90} Agent received the message: RES
97 2013-01-04 13:44:47.716: [ AGFW][1111320896] {0:6:90} Preparing CHECK command for: or
2013-01-04 13:44:47.717: [ COMMCRS][1126029632]clscsendx: (0x2aaab0245930) Physical conn
98
99 [ clsdmc][1126029632]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
10 2013-01-04 13:44:47.717: [ora.crsd][1126029632] {0:6:90} [check] Error = error 11 encoun
0 2013-01-04 13:44:47.717: [ora.crsd][1126029632] {0:6:90} [check] Calling PID check for d
10 2013-01-04 13:44:47.717: [ora.crsd][1126029632] {0:6:90} [check] Process id 19179 transl
1 2013-01-04 13:44:47.717: [ COMMCRS][1126029632]clscsendx: (0x2aaab0245930) Connection no
10
[ clsdmc][1126029632]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
2 2013-01-04 13:44:47.718: [ora.crsd][1126029632] {0:6:90} [check] Error = error 6 encount
10 2013-01-04 13:44:47.718: [ora.crsd][1126029632] {0:6:90} [check] DaemonAgent::check retu
3 2013-01-04 13:44:47.718: [ AGFW][1111320896] {0:6:90} ora.crsd 1 1 state changed from
2013-01-04 13:44:47.718: [ AGFW][1111320896] {0:6:91} Generating new Tint for unplann
10 2013-01-04 13:44:47.718: [ AGFW][1111320896] {0:6:91} Agent sending message to PE: RE
4 2013-01-04 13:44:47.725: [ AGFW][1111320896] {0:6:91} Agent received the message: RES
10 2013-01-04 13:44:47.726: [ AGFW][1111320896] {0:6:91} Preparing START command for: or
5 2013-01-04 13:44:47.726: [ AGFW][1111320896] {0:6:91} ora.crsd 1 1 state changed from
2013-01-04 13:44:47.726: [ora.crsd][1113422144] {0:6:91} [start] (:CLSN00107:) clsn_agen
10 2013-01-04 13:44:47.726: [ora.crsd][1113422144] {0:6:91} [start] Arg Value = reboot
6 2013-01-04 13:44:47.726: [ora.crsd][1113422144] {0:6:91} [start] getOracleHomeAttrib: or
10 2013-01-04 13:44:47.726: [ora.crsd][1113422144] {0:6:91} [start] Utils::getCrsHome crsHo
7 2013-01-04 13:44:47.726: [ora.crsd][1113422144] {0:6:91} [start] Utils::execCmd 1 USR_OR
10 2013-01-04 13:44:47.727: [ora.crsd][1113422144] {0:6:91} [start] Utils::getCrsHome crsHo
2013-01-04 13:44:47.727: [ora.crsd][1113422144] {0:6:91} [start] Adding Environment Vari
8 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
10 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
9 2013-01-04 13:44:47.727: [ora.crsd][1113422144] {0:6:91} [start] Adding Environment Vari
11 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
0 2013-01-04 13:44:47.727: [ora.crsd][1113422144] {0:6:91} [start] Adding Environment Vari
11 2013-01-04 13:44:47.727: [ora.crsd][1113422144] {0:6:91} [start] Adding Environment Vari
1 2013-01-04 13:44:47.727: [ora.crsd][1113422144] {0:6:91} [start] Utils:execCmd action =
11 2013-01-04 13:44:47.727: [ora.crsd][1113422144] {0:6:91} [start] getOracleHomeAttrib: or
2 2013-01-04 13:44:47.730: [ora.crsd][1113422144] {0:6:91} [start] execCmd ret = 0
2013-01-04 13:44:47.730: [ USRTHRD][1113422144] {0:6:91} Thread:[DaemonCheck:crsd]stop {
11 2013-01-04 13:44:47.730: [ USRTHRD][1113422144] {0:6:91} Thread:[DaemonCheck:crsd]stop }
3 2013-01-04 13:44:47.733: [ USRTHRD][1109219648] {0:0:113} use all detected INF
11 2013-01-04 13:44:48.732: [ora.crsd][1113422144] {0:6:91} [start] without returnbuf
4 2013-01-04 13:44:48.735: [ora.crsd][1113422144] {0:6:91} [start] clsdmc_respget return:
11 2013-01-04 13:44:48.735: [ora.crsd][1113422144] {0:6:91} [start] getOracleHomeAttrib: or
2013-01-04 13:44:48.736: [ora.crsd][1113422144] {0:6:91} [start] PID 19192 from /data/or
5 2013-01-04 13:44:48.736: [ora.crsd][1113422144] {0:6:91} [start] }DaemonAgent::start
11 2013-01-04 13:44:48.736: [ora.crsd][1113422144] {0:6:91} [start] (:CLSN00107:) clsn_agen
6 2013-01-04 13:44:48.736: [ AGFW][1113422144] {0:6:91} Command: start for resource: or
11 2013-01-04 13:44:48.737: [ AGFW][1111320896] {0:6:91} Agent sending reply for: RESOUR
2013-01-04 13:44:48.740: [ora.crsd][1113422144] {0:6:91} [check] clsdmc_respget return:
7 2013-01-04 13:44:48.740: [ USRTHRD][1113422144] {0:6:91} Thread:[DaemonCheck:crsd]start
11 2013-01-04 13:44:48.740: [ USRTHRD][1113422144] {0:6:91} Thread:[DaemonCheck:crsd]start
8 2013-01-04 13:44:48.740: [ora.crsd][1113422144] {0:6:91} [check] DaemonAgent::check retu
11 2013-01-04 13:44:48.740: [ AGFW][1111320896] {0:6:91} ora.crsd 1 1 state changed from
9 2013-01-04 13:44:48.741: [ AGFW][1111320896] {0:6:91} Started implicit monitor for [o
2013-01-04 13:44:48.741: [ AGFW][1111320896] {0:6:91} Agent sending last reply for: R
12 2013-01-04 13:44:49.727: [ USRTHRD][1109219648] {0:0:113} use all detected INF
0 2013-01-04 13:44:49.744: [ USRTHRD][1121827136] {0:6:91} Thread:[DaemonCheck:crsd]Daemon
12 2013-01-04 13:44:49.744: [ USRTHRD][1121827136] {0:6:91} Thread:[DaemonCheck:crsd]Initia
1 2013-01-04 13:44:49.744: [ USRTHRD][1121827136] {0:6:91} Check action requested by agent
2013-01-04 13:44:49.744: [ USRTHRD][1121827136] {0:6:91} Thread:[DaemonCheck:crsd]isRunn
12 2013-01-04 13:44:49.744: [ AGFW][1111320896] {0:6:91} Agent received the message: RES
2 2013-01-04 13:44:49.744: [ AGFW][1111320896] {0:6:91} Preparing CHECK command for: or
12 2013-01-04 13:44:49.745: [ COMMCRS][1113422144]clscsendx: (0x12b1f3c0) Physical connecti
3
12 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
4 2013-01-04 13:44:49.745: [ora.crsd][1113422144] {0:6:91} [check] Error = error 11 encoun
2013-01-04 13:44:49.745: [ora.crsd][1113422144] {0:6:91} [check] Calling PID check for d
12 2013-01-04 13:44:49.746: [ora.crsd][1113422144] {0:6:91} [check] Process id 19192 transl
5 2013-01-04 13:44:49.746: [ COMMCRS][1113422144]clscsendx: (0x12b1f3c0) Connection not ac
12
6 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
12 2013-01-04 13:44:49.746: [ora.crsd][1113422144] {0:6:91} [check] Error = error 6 encount
2013-01-04 13:44:49.746: [ora.crsd][1113422144] {0:6:91} [check] DaemonAgent::check retu
7 2013-01-04 13:44:49.746: [ AGFW][1111320896] {0:6:91} ora.crsd 1 1 state changed from
12 2013-01-04 13:44:49.746: [ AGFW][1111320896] {0:6:92} Generating new Tint for unplann
8 2013-01-04 13:44:49.746: [ AGFW][1111320896] {0:6:92} Agent sending message to PE: RE
12 2013-01-04 13:44:49.753: [ AGFW][1111320896] {0:6:92} Agent received the message: RES
2013-01-04 13:44:49.753: [ AGFW][1111320896] {0:6:92} Preparing START command for: or
9 2013-01-04 13:44:49.753: [ AGFW][1111320896] {0:6:92} ora.crsd 1 1 state changed from
13 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] (:CLSN00107:) clsn_agen
0 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Arg Value = reboot
13 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] getOracleHomeAttrib: or
1 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Utils::getCrsHome crsHo
2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Utils::execCmd 1 USR_OR
13 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Utils::getCrsHome crsHo
2 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Adding Environment Vari
13 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
3 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Adding Environment Vari
13 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
4 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
13 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Adding Environment Vari
5 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Adding Environment Vari
13 2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] Utils:execCmd action =
2013-01-04 13:44:49.754: [ora.crsd][1113422144] {0:6:92} [start] getOracleHomeAttrib: or
6 2013-01-04 13:44:49.759: [ora.crsd][1113422144] {0:6:92} [start] execCmd ret = 0
13 2013-01-04 13:44:49.760: [ USRTHRD][1113422144] {0:6:92} Thread:[DaemonCheck:crsd]stop {
7 2013-01-04 13:44:49.760: [ USRTHRD][1113422144] {0:6:92} Thread:[DaemonCheck:crsd]stop }
13 2013-01-04 13:44:50.761: [ora.crsd][1113422144] {0:6:92} [start] without returnbuf
8 2013-01-04 13:44:50.763: [ora.crsd][1113422144] {0:6:92} [start] clsdmc_respget return:
2013-01-04 13:44:50.763: [ora.crsd][1113422144] {0:6:92} [start] getOracleHomeAttrib: or
13 2013-01-04 13:44:50.763: [ora.crsd][1113422144] {0:6:92} [start] PID 19205 from /data/or
9 2013-01-04 13:44:50.763: [ora.crsd][1113422144] {0:6:92} [start] }DaemonAgent::start
14 2013-01-04 13:44:50.763: [ora.crsd][1113422144] {0:6:92} [start] (:CLSN00107:) clsn_agen
0 2013-01-04 13:44:50.763: [ AGFW][1113422144] {0:6:92} Command: start for resource: or
2013-01-04 13:44:50.764: [ AGFW][1111320896] {0:6:92} Agent sending reply for: RESOUR
14 2013-01-04 13:44:50.767: [ora.crsd][1113422144] {0:6:92} [check] clsdmc_respget return:
1 2013-01-04 13:44:50.768: [ USRTHRD][1113422144] {0:6:92} Thread:[DaemonCheck:crsd]start
14 2013-01-04 13:44:50.768: [ USRTHRD][1113422144] {0:6:92} Thread:[DaemonCheck:crsd]start
2 2013-01-04 13:44:50.768: [ora.crsd][1113422144] {0:6:92} [check] DaemonAgent::check retu
14 2013-01-04 13:44:50.768: [ AGFW][1111320896] {0:6:92} ora.crsd 1 1 state changed from
2013-01-04 13:44:50.768: [ AGFW][1111320896] {0:6:92} Started implicit monitor for [o
3 2013-01-04 13:44:50.768: [ AGFW][1111320896] {0:6:92} Agent sending last reply for: R
14 2013-01-04 13:44:51.731: [ USRTHRD][1109219648] {0:0:113} use all detected INF
4 2013-01-04 13:44:51.771: [ USRTHRD][1119725888] {0:6:92} Thread:[DaemonCheck:crsd]Daemon
14 2013-01-04 13:44:51.771: [ USRTHRD][1119725888] {0:6:92} Thread:[DaemonCheck:crsd]Initia
2013-01-04 13:44:51.771: [ USRTHRD][1119725888] {0:6:92} Check action requested by agent
5 2013-01-04 13:44:51.771: [ USRTHRD][1119725888] {0:6:92} Thread:[DaemonCheck:crsd]isRunn
14 2013-01-04 13:44:51.771: [ AGFW][1111320896] {0:6:92} Agent received the message: RES
6 2013-01-04 13:44:51.771: [ AGFW][1111320896] {0:6:92} Preparing CHECK command for: or
14 2013-01-04 13:44:51.772: [ COMMCRS][1113422144]clscsendx: (0x2aaab00733b0) Physical conn
7
14 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
2013-01-04 13:44:51.772: [ora.crsd][1113422144] {0:6:92} [check] Error = error 11 encoun
8 2013-01-04 13:44:51.773: [ora.crsd][1113422144] {0:6:92} [check] Calling PID check for d
14 2013-01-04 13:44:51.773: [ora.crsd][1113422144] {0:6:92} [check] Process id 19205 transl
9 2013-01-04 13:44:51.773: [ COMMCRS][1113422144]clscsendx: (0x2aaab00733b0) Connection no
15
[ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
0 2013-01-04 13:44:51.773: [ora.crsd][1113422144] {0:6:92} [check] Error = error 6 encount
15 2013-01-04 13:44:51.773: [ora.crsd][1113422144] {0:6:92} [check] DaemonAgent::check retu
1 2013-01-04 13:44:51.773: [ AGFW][1111320896] {0:6:92} ora.crsd 1 1 state changed from
15 2013-01-04 13:44:51.773: [ AGFW][1111320896] {0:6:93} Generating new Tint for unplann
2013-01-04 13:44:51.773: [ AGFW][1111320896] {0:6:93} Agent sending message to PE: RE
2 2013-01-04 13:44:51.780: [ AGFW][1111320896] {0:6:93} Agent received the message: RES
15 2013-01-04 13:44:51.780: [ AGFW][1111320896] {0:6:93} Preparing START command for: or
3 2013-01-04 13:44:51.780: [ AGFW][1111320896] {0:6:93} ora.crsd 1 1 state changed from
15 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] (:CLSN00107:) clsn_agen
4 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] Arg Value = reboot
2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] getOracleHomeAttrib: or
15 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] Utils::getCrsHome crsHo
5 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] Utils::execCmd 1 USR_OR
15 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] Utils::getCrsHome crsHo
6 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] Adding Environment Vari
ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
15 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
7 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] Adding Environment Vari
15 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
8 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
15 2013-01-04 13:44:51.781: [ora.crsd][1113422144] {0:6:93} [start] Adding Environment Vari
2013-01-04 13:44:51.782: [ora.crsd][1113422144] {0:6:93} [start] Adding Environment Vari
9 2013-01-04 13:44:51.782: [ora.crsd][1113422144] {0:6:93} [start] Utils:execCmd action =
16 2013-01-04 13:44:51.782: [ora.crsd][1113422144] {0:6:93} [start] getOracleHomeAttrib: or
0 2013-01-04 13:44:51.784: [ora.crsd][1113422144] {0:6:93} [start] execCmd ret = 0
16 2013-01-04 13:44:51.785: [ USRTHRD][1113422144] {0:6:93} Thread:[DaemonCheck:crsd]stop {
1 2013-01-04 13:44:51.785: [ USRTHRD][1113422144] {0:6:93} Thread:[DaemonCheck:crsd]stop }
2013-01-04 13:44:52.788: [ora.crsd][1113422144] {0:6:93} [start] without returnbuf
16 2013-01-04 13:44:52.792: [ora.crsd][1113422144] {0:6:93} [start] clsdmc_respget return:
2 2013-01-04 13:44:52.792: [ora.crsd][1113422144] {0:6:93} [start] getOracleHomeAttrib: or
16 2013-01-04 13:44:52.792: [ora.crsd][1113422144] {0:6:93} [start] PID 19218 from /data/or
3 2013-01-04 13:44:52.792: [ora.crsd][1113422144] {0:6:93} [start] }DaemonAgent::start
2013-01-04 13:44:52.792: [ora.crsd][1113422144] {0:6:93} [start] (:CLSN00107:) clsn_agen
16 2013-01-04 13:44:52.793: [ AGFW][1113422144] {0:6:93} Command: start for resource: or
4 2013-01-04 13:44:52.793: [ AGFW][1111320896] {0:6:93} Agent sending reply for: RESOUR
16 2013-01-04 13:44:52.794: [ora.crsd][1113422144] {0:6:93} [check] clsdmc_respget return:
5 2013-01-04 13:44:52.796: [ USRTHRD][1113422144] {0:6:93} Thread:[DaemonCheck:crsd]start
16 2013-01-04 13:44:52.796: [ USRTHRD][1113422144] {0:6:93} Thread:[DaemonCheck:crsd]start
2013-01-04 13:44:52.796: [ora.crsd][1113422144] {0:6:93} [check] DaemonAgent::check retu
6 2013-01-04 13:44:52.796: [ AGFW][1111320896] {0:6:93} ora.crsd 1 1 state changed from
16 2013-01-04 13:44:52.797: [ AGFW][1111320896] {0:6:93} Started implicit monitor for [o
7 2013-01-04 13:44:52.797: [ AGFW][1111320896] {0:6:93} Agent sending last reply for: R
16 2013-01-04 13:44:53.727: [ USRTHRD][1109219648] {0:0:113} use all detected INF
2013-01-04 13:44:53.800: [ USRTHRD][1107118400] {0:6:93} Thread:[DaemonCheck:crsd]Daemon
8 2013-01-04 13:44:53.800: [ USRTHRD][1107118400] {0:6:93} Thread:[DaemonCheck:crsd]Initia
16 2013-01-04 13:44:53.800: [ USRTHRD][1107118400] {0:6:93} Check action requested by agent
9 2013-01-04 13:44:53.800: [ USRTHRD][1107118400] {0:6:93} Thread:[DaemonCheck:crsd]isRunn
17 2013-01-04 13:44:53.800: [ AGFW][1111320896] {0:6:93} Agent received the message: RES
0 2013-01-04 13:44:53.801: [ AGFW][1111320896] {0:6:93} Preparing CHECK command for: or
2013-01-04 13:44:53.801: [ COMMCRS][1113422144]clscsendx: (0x2aaab006bb30) Physical conn
17
1 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
17 2013-01-04 13:44:53.801: [ora.crsd][1113422144] {0:6:93} [check] Error = error 11 encoun
2 2013-01-04 13:44:53.801: [ora.crsd][1113422144] {0:6:93} [check] Calling PID check for d
2013-01-04 13:44:53.802: [ora.crsd][1113422144] {0:6:93} [check] Process id 19218 transl
17 2013-01-04 13:44:53.802: [ COMMCRS][1113422144]clscsendx: (0x2aaab006bb30) Connection no
3
17 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
4 2013-01-04 13:44:53.802: [ora.crsd][1113422144] {0:6:93} [check] Error = error 6 encount
17 2013-01-04 13:44:53.802: [ora.crsd][1113422144] {0:6:93} [check] DaemonAgent::check retu
2013-01-04 13:44:53.802: [ AGFW][1111320896] {0:6:93} ora.crsd 1 1 state changed from
5 2013-01-04 13:44:53.802: [ AGFW][1111320896] {0:6:94} Generating new Tint for unplann
17 2013-01-04 13:44:53.802: [ AGFW][1111320896] {0:6:94} Agent sending message to PE: RE
6 2013-01-04 13:44:53.809: [ AGFW][1111320896] {0:6:94} Agent received the message: RES
17 2013-01-04 13:44:53.809: [ AGFW][1111320896] {0:6:94} Preparing START command for: or
7 2013-01-04 13:44:53.809: [ AGFW][1111320896] {0:6:94} ora.crsd 1 1 state changed from
2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] (:CLSN00107:) clsn_agen
17 2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] Arg Value = reboot
8 2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] getOracleHomeAttrib: or
17 2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] Utils::getCrsHome crsHo
9 2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] Utils::execCmd 1 USR_OR
2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] Utils::getCrsHome crsHo
18 2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] Adding Environment Vari
0 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
18 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
1 2013-01-04 13:44:53.810: [ora.crsd][1113422144] {0:6:94} [start] Adding Environment Vari
18 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
2 2013-01-04 13:44:53.811: [ora.crsd][1113422144] {0:6:94} [start] Adding Environment Vari
18 2013-01-04 13:44:53.811: [ora.crsd][1113422144] {0:6:94} [start] Adding Environment Vari
3 2013-01-04 13:44:53.811: [ora.crsd][1113422144] {0:6:94} [start] Utils:execCmd action =
18 2013-01-04 13:44:53.811: [ora.crsd][1113422144] {0:6:94} [start] getOracleHomeAttrib: or
4 2013-01-04 13:44:53.813: [ora.crsd][1113422144] {0:6:94} [start] execCmd ret = 0
2013-01-04 13:44:53.815: [ USRTHRD][1113422144] {0:6:94} Thread:[DaemonCheck:crsd]stop {
18 2013-01-04 13:44:53.815: [ USRTHRD][1113422144] {0:6:94} Thread:[DaemonCheck:crsd]stop }
5 [ clsdmc][1123928384]CLSDMC.C returnbuflen=8, extraDataBuf=EE, returnbuf=12CA3BB0
18 2013-01-04 13:44:53.877: [ora.ctssd][1123928384] {0:0:2} [check] clsdmc_respget return:
6 2013-01-04 13:44:53.877: [ora.ctssd][1123928384] {0:0:2} [check] translateReturnCodes, r
2013-01-04 13:44:54.816: [ora.crsd][1113422144] {0:6:94} [start] without returnbuf
18 2013-01-04 13:44:54.818: [ora.crsd][1113422144] {0:6:94} [start] clsdmc_respget return:
7 2013-01-04 13:44:54.818: [ora.crsd][1113422144] {0:6:94} [start] getOracleHomeAttrib: or
18 2013-01-04 13:44:54.818: [ora.crsd][1113422144] {0:6:94} [start] PID 19232 from /data/or
8 2013-01-04 13:44:54.818: [ora.crsd][1113422144] {0:6:94} [start] }DaemonAgent::start
18 2013-01-04 13:44:54.818: [ora.crsd][1113422144] {0:6:94} [start] (:CLSN00107:) clsn_agen
2013-01-04 13:44:54.818: [ AGFW][1113422144] {0:6:94} Command: start for resource: or
9 2013-01-04 13:44:54.819: [ AGFW][1111320896] {0:6:94} Agent sending reply for: RESOUR
19 2013-01-04 13:44:54.821: [ora.crsd][1113422144] {0:6:94} [check] clsdmc_respget return:
0 2013-01-04 13:44:54.822: [ USRTHRD][1113422144] {0:6:94} Thread:[DaemonCheck:crsd]start
19 2013-01-04 13:44:54.822: [ USRTHRD][1113422144] {0:6:94} Thread:[DaemonCheck:crsd]start
2013-01-04 13:44:54.822: [ora.crsd][1113422144] {0:6:94} [check] DaemonAgent::check retu
1 2013-01-04 13:44:54.822: [ AGFW][1111320896] {0:6:94} ora.crsd 1 1 state changed from
19 2013-01-04 13:44:54.822: [ AGFW][1111320896] {0:6:94} Started implicit monitor for [o
2 2013-01-04 13:44:54.822: [ AGFW][1111320896] {0:6:94} Agent sending last reply for: R
19 2013-01-04 13:44:55.731: [ USRTHRD][1109219648] {0:0:113} use all detected INF
3 2013-01-04 13:44:55.824: [ USRTHRD][1121827136] {0:6:94} Thread:[DaemonCheck:crsd]Daemon
2013-01-04 13:44:55.824: [ USRTHRD][1121827136] {0:6:94} Thread:[DaemonCheck:crsd]Initia
19 2013-01-04 13:44:55.824: [ USRTHRD][1121827136] {0:6:94} Check action requested by agent
4 2013-01-04 13:44:55.824: [ USRTHRD][1121827136] {0:6:94} Thread:[DaemonCheck:crsd]isRunn
19 2013-01-04 13:44:55.824: [ AGFW][1111320896] {0:6:94} Agent received the message: RES
5 2013-01-04 13:44:55.824: [ AGFW][1111320896] {0:6:94} Preparing CHECK command for: or
2013-01-04 13:44:55.825: [ COMMCRS][1123928384]clscsendx: (0x12c9fa90) Physical connecti
19
6 [ clsdmc][1123928384]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
19 2013-01-04 13:44:55.825: [ora.crsd][1123928384] {0:6:94} [check] Error = error 11 encoun
7 2013-01-04 13:44:55.825: [ora.crsd][1123928384] {0:6:94} [check] Calling PID check for d
19 2013-01-04 13:44:55.825: [ora.crsd][1123928384] {0:6:94} [check] Process id 19232 transl
2013-01-04 13:44:55.825: [ COMMCRS][1123928384]clscsendx: (0x12c9fa90) Connection not ac
8
19 [ clsdmc][1123928384]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
9 2013-01-04 13:44:55.826: [ora.crsd][1123928384] {0:6:94} [check] Error = error 6 encount
20 2013-01-04 13:44:55.826: [ora.crsd][1123928384] {0:6:94} [check] DaemonAgent::check retu
0 2013-01-04 13:44:55.826: [ AGFW][1111320896] {0:6:94} ora.crsd 1 1 state changed from
20 2013-01-04 13:44:55.826: [ AGFW][1111320896] {0:6:95} Generating new Tint for unplann
2013-01-04 13:44:55.826: [ AGFW][1111320896] {0:6:95} Agent sending message to PE: RE
1 2013-01-04 13:44:55.833: [ AGFW][1111320896] {0:6:95} Agent received the message: RES
20 2013-01-04 13:44:55.833: [ AGFW][1111320896] {0:6:95} Preparing START command for: or
2 2013-01-04 13:44:55.833: [ AGFW][1111320896] {0:6:95} ora.crsd 1 1 state changed from
20 2013-01-04 13:44:55.834: [ora.crsd][1113422144] {0:6:95} [start] (:CLSN00107:) clsn_agen
2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Arg Value = reboot
3 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] getOracleHomeAttrib: or
20 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Utils::getCrsHome crsHo
4 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Utils::execCmd 1 USR_OR
20 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Utils::getCrsHome crsHo
5 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Adding Environment Vari
ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
20 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
6 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Adding Environment Vari
20 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
7 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
20 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Adding Environment Vari
2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] Utils:execCmd action =
8 2013-01-04 13:44:55.835: [ora.crsd][1113422144] {0:6:95} [start] getOracleHomeAttrib: or
20 2013-01-04 13:44:55.838: [ora.crsd][1113422144] {0:6:95} [start] execCmd ret = 0
9 2013-01-04 13:44:55.838: [ USRTHRD][1113422144] {0:6:95} Thread:[DaemonCheck:crsd]stop {
21 2013-01-04 13:44:55.838: [ USRTHRD][1113422144] {0:6:95} Thread:[DaemonCheck:crsd]stop }
2013-01-04 13:44:56.839: [ora.crsd][1113422144] {0:6:95} [start] without returnbuf
0 2013-01-04 13:44:56.841: [ora.crsd][1113422144] {0:6:95} [start] clsdmc_respget return:
21 2013-01-04 13:44:56.841: [ora.crsd][1113422144] {0:6:95} [start] getOracleHomeAttrib: or
1 2013-01-04 13:44:56.841: [ora.crsd][1113422144] {0:6:95} [start] PID 19245 from /data/or
21 2013-01-04 13:44:56.841: [ora.crsd][1113422144] {0:6:95} [start] }DaemonAgent::start
2 2013-01-04 13:44:56.841: [ora.crsd][1113422144] {0:6:95} [start] (:CLSN00107:) clsn_agen
2013-01-04 13:44:56.841: [ AGFW][1113422144] {0:6:95} Command: start for resource: or
21 2013-01-04 13:44:56.842: [ AGFW][1111320896] {0:6:95} Agent sending reply for: RESOUR
3 2013-01-04 13:44:56.844: [ora.crsd][1113422144] {0:6:95} [check] clsdmc_respget return:
21 2013-01-04 13:44:56.845: [ USRTHRD][1113422144] {0:6:95} Thread:[DaemonCheck:crsd]start
4 2013-01-04 13:44:56.845: [ USRTHRD][1113422144] {0:6:95} Thread:[DaemonCheck:crsd]start
2013-01-04 13:44:56.845: [ora.crsd][1113422144] {0:6:95} [check] DaemonAgent::check retu
21 2013-01-04 13:44:56.846: [ AGFW][1111320896] {0:6:95} ora.crsd 1 1 state changed from
5 2013-01-04 13:44:56.846: [ AGFW][1111320896] {0:6:95} Started implicit monitor for [o
21 2013-01-04 13:44:56.846: [ AGFW][1111320896] {0:6:95} Agent sending last reply for: R
6 2013-01-04 13:44:57.019: [ora.crf][1113422144] {0:0:2} [check] clsdmc_respget return: st
21 2013-01-04 13:44:57.019: [ora.crf][1113422144] {0:0:2} [check] Check return = 0, state d
2013-01-04 13:44:57.737: [ USRTHRD][1109219648] {0:0:113} use all detected INF
7 2013-01-04 13:44:57.848: [ USRTHRD][1119725888] {0:6:95} Thread:[DaemonCheck:crsd]Daemon
21 2013-01-04 13:44:57.848: [ USRTHRD][1119725888] {0:6:95} Thread:[DaemonCheck:crsd]Initia
8 2013-01-04 13:44:57.848: [ USRTHRD][1119725888] {0:6:95} Check action requested by agent
2013-01-04 13:44:57.848: [ USRTHRD][1119725888] {0:6:95} Thread:[DaemonCheck:crsd]isRunn
21 2013-01-04 13:44:57.848: [ AGFW][1111320896] {0:6:95} Agent received the message: RES
9 2013-01-04 13:44:57.849: [ AGFW][1111320896] {0:6:95} Preparing CHECK command for: or
22 2013-01-04 13:44:57.849: [ COMMCRS][1113422144]clscsendx: (0x12b9a890) Physical connecti
0
22 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
2013-01-04 13:44:57.849: [ora.crsd][1113422144] {0:6:95} [check] Error = error 11 encoun
1 2013-01-04 13:44:57.849: [ora.crsd][1113422144] {0:6:95} [check] Calling PID check for d
22 2013-01-04 13:44:57.849: [ora.crsd][1113422144] {0:6:95} [check] Process id 19245 transl
2 2013-01-04 13:44:57.849: [ COMMCRS][1113422144]clscsendx: (0x12b9a890) Connection not ac
22
3 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
22 2013-01-04 13:44:57.850: [ora.crsd][1113422144] {0:6:95} [check] Error = error 6 encount
2013-01-04 13:44:57.850: [ora.crsd][1113422144] {0:6:95} [check] DaemonAgent::check retu
4 2013-01-04 13:44:57.850: [ AGFW][1111320896] {0:6:95} ora.crsd 1 1 state changed from
22 2013-01-04 13:44:57.850: [ AGFW][1111320896] {0:6:96} Generating new Tint for unplann
5 2013-01-04 13:44:57.850: [ AGFW][1111320896] {0:6:96} Agent sending message to PE: RE
22 2013-01-04 13:44:57.857: [ AGFW][1111320896] {0:6:96} Agent received the message: RES
2013-01-04 13:44:57.857: [ AGFW][1111320896] {0:6:96} Preparing START command for: or
6 2013-01-04 13:44:57.857: [ AGFW][1111320896] {0:6:96} ora.crsd 1 1 state changed from
22 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] (:CLSN00107:) clsn_agen
7 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Arg Value = reboot
22 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] getOracleHomeAttrib: or
8 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Utils::getCrsHome crsHo
2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Utils::execCmd 1 USR_OR
22 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Utils::getCrsHome crsHo
9 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Adding Environment Vari
23 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
0 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
23 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Adding Environment Vari
ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
1 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
23 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Adding Environment Vari
2 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Adding Environment Vari
23 2013-01-04 13:44:57.859: [ora.crsd][1126029632] {0:6:96} [start] Utils:execCmd action =
2013-01-04 13:44:57.860: [ora.crsd][1126029632] {0:6:96} [start] getOracleHomeAttrib: or
3 2013-01-04 13:44:57.863: [ora.crsd][1126029632] {0:6:96} [start] execCmd ret = 0
23 2013-01-04 13:44:57.863: [ USRTHRD][1126029632] {0:6:96} Thread:[DaemonCheck:crsd]stop {
4 2013-01-04 13:44:57.863: [ USRTHRD][1126029632] {0:6:96} Thread:[DaemonCheck:crsd]stop }
23 2013-01-04 13:44:58.864: [ora.crsd][1126029632] {0:6:96} [start] without returnbuf
5 2013-01-04 13:44:58.866: [ora.crsd][1126029632] {0:6:96} [start] clsdmc_respget return:
2013-01-04 13:44:58.866: [ora.crsd][1126029632] {0:6:96} [start] getOracleHomeAttrib: or
23 2013-01-04 13:44:58.866: [ora.crsd][1126029632] {0:6:96} [start] PID 19259 from /data/or
6 2013-01-04 13:44:58.866: [ora.crsd][1126029632] {0:6:96} [start] }DaemonAgent::start
23 2013-01-04 13:44:58.866: [ora.crsd][1126029632] {0:6:96} [start] (:CLSN00107:) clsn_agen
7 2013-01-04 13:44:58.867: [ AGFW][1126029632] {0:6:96} Command: start for resource: or
2013-01-04 13:44:58.867: [ AGFW][1111320896] {0:6:96} Agent sending reply for: RESOUR
23 2013-01-04 13:44:58.870: [ora.crsd][1126029632] {0:6:96} [check] clsdmc_respget return:
8 2013-01-04 13:44:58.871: [ USRTHRD][1126029632] {0:6:96} Thread:[DaemonCheck:crsd]start
23 2013-01-04 13:44:58.871: [ USRTHRD][1126029632] {0:6:96} Thread:[DaemonCheck:crsd]start
9 2013-01-04 13:44:58.871: [ora.crsd][1126029632] {0:6:96} [check] DaemonAgent::check retu
24 2013-01-04 13:44:58.871: [ AGFW][1111320896] {0:6:96} ora.crsd 1 1 state changed from
2013-01-04 13:44:58.871: [ AGFW][1111320896] {0:6:96} Started implicit monitor for [o
0 2013-01-04 13:44:58.871: [ AGFW][1111320896] {0:6:96} Agent sending last reply for: R
24 2013-01-04 13:44:59.742: [ USRTHRD][1109219648] {0:0:113} use all detected INF
1 2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] Arg Value = chec
2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] getOracleHomeAtt
24 2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] Utils::getCrsHom
2 2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] Utils::execCmd 1
24 2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] Utils::getCrsHom
3 2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] Adding Environme
2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] Adding Environme
24 2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] Utils:execCmd ac
4 2013-01-04 13:44:59.814: [ora.drivers.acfs][1126029632] {0:0:2} [check] getOracleHomeAtt
24 2013-01-04 13:44:59.875: [ USRTHRD][1107118400] {0:6:96} Thread:[DaemonCheck:crsd]Daemon
5 2013-01-04 13:44:59.875: [ USRTHRD][1107118400] {0:6:96} Thread:[DaemonCheck:crsd]Initia
24 2013-01-04 13:44:59.875: [ USRTHRD][1107118400] {0:6:96} Check action requested by agent
2013-01-04 13:44:59.876: [ USRTHRD][1107118400] {0:6:96} Thread:[DaemonCheck:crsd]isRunn
6 2013-01-04 13:44:59.876: [ AGFW][1111320896] {0:6:96} Agent received the message: RES
24 2013-01-04 13:44:59.876: [ AGFW][1111320896] {0:6:96} Preparing CHECK command for: or
7 2013-01-04 13:44:59.877: [ COMMCRS][1113422144]clscsendx: (0x12c9fa90) Physical connecti
24
8 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
2013-01-04 13:44:59.877: [ora.crsd][1113422144] {0:6:96} [check] Error = error 11 encoun
24 2013-01-04 13:44:59.877: [ora.crsd][1113422144] {0:6:96} [check] Calling PID check for d
9 2013-01-04 13:44:59.877: [ora.crsd][1113422144] {0:6:96} [check] Process id 19259 transl
25 2013-01-04 13:44:59.877: [ COMMCRS][1113422144]clscsendx: (0x12c9fa90) Connection not ac
0
25 [ clsdmc][1113422144]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
1 2013-01-04 13:44:59.877: [ora.crsd][1113422144] {0:6:96} [check] Error = error 6 encount
2013-01-04 13:44:59.877: [ora.crsd][1113422144] {0:6:96} [check] DaemonAgent::check retu
25 2013-01-04 13:44:59.878: [ AGFW][1111320896] {0:6:96} ora.crsd 1 1 state changed from
2 2013-01-04 13:44:59.878: [ AGFW][1111320896] {0:6:97} Generating new Tint for unplann
25 2013-01-04 13:44:59.878: [ AGFW][1111320896] {0:6:97} Agent sending message to PE: RE
3 2013-01-04 13:44:59.885: [ AGFW][1111320896] {0:6:97} Agent received the message: RES
25 2013-01-04 13:44:59.885: [ AGFW][1111320896] {0:6:97} Preparing START command for: or
2013-01-04 13:44:59.885: [ AGFW][1111320896] {0:6:97} ora.crsd 1 1 state changed from
4 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] (:CLSN00107:) clsn_agen
25 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Arg Value = reboot
5 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] getOracleHomeAttrib: or
25 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Utils::getCrsHome crsHo
2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Utils::execCmd 1 USR_OR
6 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Utils::getCrsHome crsHo
25 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Adding Environment Vari
7 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
25 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
8 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Adding Environment Vari
ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
25 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
9 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Adding Environment Vari
26 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Adding Environment Vari
0 2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] Utils:execCmd action =
2013-01-04 13:44:59.886: [ora.crsd][1113422144] {0:6:97} [start] getOracleHomeAttrib: or
26 2013-01-04 13:44:59.889: [ora.crsd][1113422144] {0:6:97} [start] execCmd ret = 0
1 2013-01-04 13:44:59.890: [ USRTHRD][1113422144] {0:6:97} Thread:[DaemonCheck:crsd]stop {
26 2013-01-04 13:44:59.890: [ USRTHRD][1113422144] {0:6:97} Thread:[DaemonCheck:crsd]stop }
2 2013-01-04 13:44:59.919: [ora.drivers.acfs][1126029632] {0:0:2} [check] execCmd ret = 0
26 2013-01-04 13:45:00.891: [ora.crsd][1113422144] {0:6:97} [start] without returnbuf
2013-01-04 13:45:00.893: [ora.crsd][1113422144] {0:6:97} [start] clsdmc_respget return:
3 2013-01-04 13:45:00.893: [ora.crsd][1113422144] {0:6:97} [start] getOracleHomeAttrib: or
26 2013-01-04 13:45:00.893: [ora.crsd][1113422144] {0:6:97} [start] PID 19277 from /data/or
4 2013-01-04 13:45:00.893: [ora.crsd][1113422144] {0:6:97} [start] }DaemonAgent::start
2013-01-04 13:45:00.893: [ora.crsd][1113422144] {0:6:97} [start] (:CLSN00107:) clsn_agen
26 2013-01-04 13:45:00.893: [ AGFW][1113422144] {0:6:97} Command: start for resource: or
5 2013-01-04 13:45:00.894: [ AGFW][1111320896] {0:6:97} Agent sending reply for: RESOUR
26 2013-01-04 13:45:00.898: [ora.crsd][1113422144] {0:6:97} [check] clsdmc_respget return:
6 2013-01-04 13:45:00.898: [ USRTHRD][1113422144] {0:6:97} Thread:[DaemonCheck:crsd]start
2013-01-04 13:45:00.898: [ USRTHRD][1113422144] {0:6:97} Thread:[DaemonCheck:crsd]start
26 2013-01-04 13:45:00.898: [ora.crsd][1113422144] {0:6:97} [check] DaemonAgent::check retu
7 2013-01-04 13:45:00.899: [ AGFW][1111320896] {0:6:97} ora.crsd 1 1 state changed from
26 2013-01-04 13:45:00.899: [ AGFW][1111320896] {0:6:97} Started implicit monitor for [o
8 2013-01-04 13:45:00.899: [ AGFW][1111320896] {0:6:97} Agent sending last reply for: R
26 2013-01-04 13:45:01.746: [ USRTHRD][1109219648] {0:0:113} use all detected INF
2013-01-04 13:45:01.901: [ USRTHRD][1121827136] {0:6:97} Thread:[DaemonCheck:crsd]Daemon
9 2013-01-04 13:45:01.901: [ USRTHRD][1121827136] {0:6:97} Thread:[DaemonCheck:crsd]Initia
27 2013-01-04 13:45:01.901: [ USRTHRD][1121827136] {0:6:97} Check action requested by agent
0 2013-01-04 13:45:01.901: [ USRTHRD][1121827136] {0:6:97} Thread:[DaemonCheck:crsd]isRunn
27 2013-01-04 13:45:01.901: [ AGFW][1111320896] {0:6:97} Agent received the message: RES
2013-01-04 13:45:01.901: [ AGFW][1111320896] {0:6:97} Preparing CHECK command for: or
1 2013-01-04 13:45:01.902: [ COMMCRS][1123928384]clscsendx: (0x12b9a890) Physical connecti
27
2 [ clsdmc][1123928384]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
27 2013-01-04 13:45:01.902: [ora.crsd][1123928384] {0:6:97} [check] Error = error 11 encoun
3 2013-01-04 13:45:01.902: [ora.crsd][1123928384] {0:6:97} [check] Calling PID check for d
27 2013-01-04 13:45:01.903: [ora.crsd][1123928384] {0:6:97} [check] Process id 19277 transl
2013-01-04 13:45:01.903: [ COMMCRS][1123928384]clscsendx: (0x12b9a890) Connection not ac
4
27 [ clsdmc][1123928384]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
5 2013-01-04 13:45:01.903: [ora.crsd][1123928384] {0:6:97} [check] Error = error 6 encount
27 2013-01-04 13:45:01.903: [ora.crsd][1123928384] {0:6:97} [check] DaemonAgent::check retu
6 2013-01-04 13:45:01.903: [ AGFW][1111320896] {0:6:97} ora.crsd 1 1 state changed from
27 2013-01-04 13:45:01.903: [ AGFW][1111320896] {0:6:98} Generating new Tint for unplann
2013-01-04 13:45:01.903: [ AGFW][1111320896] {0:6:98} Agent sending message to PE: RE
7 2013-01-04 13:45:01.910: [ AGFW][1111320896] {0:6:98} Agent received the message: RES
27 2013-01-04 13:45:01.910: [ AGFW][1111320896] {0:6:98} Preparing START command for: or
8 2013-01-04 13:45:01.910: [ AGFW][1111320896] {0:6:98} ora.crsd 1 1 state changed from
27 2013-01-04 13:45:01.911: [ora.crsd][1123928384] {0:6:98} [start] (:CLSN00107:) clsn_agen
2013-01-04 13:45:01.911: [ora.crsd][1123928384] {0:6:98} [start] Arg Value = reboot
9 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] getOracleHomeAttrib: or
28 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Utils::getCrsHome crsHo
0 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Utils::execCmd 1 USR_OR
28 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Utils::getCrsHome crsHo
1 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Adding Environment Vari
ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
28 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
2 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Adding Environment Vari
28 ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
3 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Adding Environment Vari
28 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Adding Environment Vari
4 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] Utils:execCmd action =
28 2013-01-04 13:45:01.912: [ora.crsd][1123928384] {0:6:98} [start] getOracleHomeAttrib: or
5 2013-01-04 13:45:01.915: [ora.crsd][1123928384] {0:6:98} [start] execCmd ret = 0
28 2013-01-04 13:45:01.916: [ USRTHRD][1123928384] {0:6:98} Thread:[DaemonCheck:crsd]stop {
2013-01-04 13:45:01.916: [ USRTHRD][1123928384] {0:6:98} Thread:[DaemonCheck:crsd]stop }
6 2013-01-04 13:45:02.917: [ora.crsd][1123928384] {0:6:98} [start] without returnbuf
28 2013-01-04 13:45:02.920: [ora.crsd][1123928384] {0:6:98} [start] clsdmc_respget return:
7 2013-01-04 13:45:02.920: [ora.crsd][1123928384] {0:6:98} [start] getOracleHomeAttrib: or
2013-01-04 13:45:02.921: [ora.crsd][1123928384] {0:6:98} [start] PID 19290 from /data/or
28 2013-01-04 13:45:02.921: [ora.crsd][1123928384] {0:6:98} [start] }DaemonAgent::start
8 2013-01-04 13:45:02.921: [ora.crsd][1123928384] {0:6:98} [start] (:CLSN00107:) clsn_agen
28 2013-01-04 13:45:02.921: [ AGFW][1123928384] {0:6:98} Command: start for resource: or
9 2013-01-04 13:45:02.922: [ AGFW][1111320896] {0:6:98} Agent sending reply for: RESOUR
2013-01-04 13:45:02.923: [ora.crsd][1123928384] {0:6:98} [check] clsdmc_respget return:
29 2013-01-04 13:45:02.924: [ USRTHRD][1123928384] {0:6:98} Thread:[DaemonCheck:crsd]start
0 2013-01-04 13:45:02.924: [ USRTHRD][1123928384] {0:6:98} Thread:[DaemonCheck:crsd]start
29 2013-01-04 13:45:02.924: [ora.crsd][1123928384] {0:6:98} [check] DaemonAgent::check retu
1 2013-01-04 13:45:02.924: [ AGFW][1111320896] {0:6:98} ora.crsd 1 1 state changed from
29 2013-01-04 13:45:02.924: [ AGFW][1111320896] {0:6:98} Started implicit monitor for [o
2013-01-04 13:45:02.925: [ AGFW][1111320896] {0:6:98} Agent sending last reply for: R
2 2013-01-04 13:45:03.752: [ USRTHRD][1109219648] {0:0:113} use all detected INF
29 2013-01-04 13:45:03.925: [ USRTHRD][1119725888] {0:6:98} Thread:[DaemonCheck:crsd]Daemon
3 2013-01-04 13:45:03.925: [ USRTHRD][1119725888] {0:6:98} Thread:[DaemonCheck:crsd]Initia
29 2013-01-04 13:45:03.925: [ USRTHRD][1119725888] {0:6:98} Check action requested by agent
2013-01-04 13:45:03.925: [ USRTHRD][1119725888] {0:6:98} Thread:[DaemonCheck:crsd]isRunn
4 2013-01-04 13:45:03.926: [ AGFW][1111320896] {0:6:98} Agent received the message: RES
29 2013-01-04 13:45:03.926: [ AGFW][1111320896] {0:6:98} Preparing CHECK command for: or
5 2013-01-04 13:45:03.926: [ COMMCRS][1126029632]clscsendx: (0x2aaab038ded0) Physical conn
29
6 [ clsdmc][1126029632]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
29 2013-01-04 13:45:03.926: [ora.crsd][1126029632] {0:6:98} [check] Error = error 11 encoun
2013-01-04 13:45:03.927: [ora.crsd][1126029632] {0:6:98} [check] Calling PID check for d
7 2013-01-04 13:45:03.927: [ora.crsd][1126029632] {0:6:98} [check] Process id 19290 transl
29 2013-01-04 13:45:03.927: [ COMMCRS][1126029632]clscsendx: (0x2aaab038ded0) Connection no
8
29 [ clsdmc][1126029632]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
9 2013-01-04 13:45:03.927: [ora.crsd][1126029632] {0:6:98} [check] Error = error 6 encount
30 2013-01-04 13:45:03.927: [ora.crsd][1126029632] {0:6:98} [check] DaemonAgent::check retu
2013-01-04 13:45:03.927: [ AGFW][1111320896] {0:6:98} ora.crsd 1 1 state changed from
0 2013-01-04 13:45:03.927: [ AGFW][1111320896] {0:6:99} Generating new Tint for unplann
30 2013-01-04 13:45:03.927: [ AGFW][1111320896] {0:6:99} Agent sending message to PE: RE
1 2013-01-04 13:45:03.935: [ AGFW][1111320896] {0:6:99} Agent received the message: RES
30 2013-01-04 13:45:03.935: [ AGFW][1111320896] {0:6:99} Preparing START command for: or
2013-01-04 13:45:03.935: [ AGFW][1111320896] {0:6:99} ora.crsd 1 1 state changed from
2 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] (:CLSN00107:) clsn_agen
30 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Arg Value = reboot
3 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] getOracleHomeAttrib: or
30 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Utils::getCrsHome crsHo
4 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Utils::execCmd 1 USR_OR
2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Utils::getCrsHome crsHo
30 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Adding Environment Vari
5 ORA_DAEMON_LOGGING_LEVELS=AGENT=1,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
30 es=1,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
6 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Adding Environment Vari
ORA_DAEMON_TRACING_LEVELS=AGENT=0,AGFW=0,CLSFRAME=0,CLSVER=0,CLUCLS=0,COMMCRS=0,COMMNS=0
30 es=0,UiServer=0,OCRAPI=1,OCRCLI=1,OCRSRV=1,OCRMAS=1,OCRMSG=1,OCRCAC=1,OCRRAW=1,OCRUTL=1,
7 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Adding Environment Vari
30 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Adding Environment Vari
8 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] Utils:execCmd action =
30 2013-01-04 13:45:03.937: [ora.crsd][1123928384] {0:6:99} [start] getOracleHomeAttrib: or
2013-01-04 13:45:03.940: [ora.crsd][1123928384] {0:6:99} [start] execCmd ret = 0
9 2013-01-04 13:45:03.941: [ USRTHRD][1123928384] {0:6:99} Thread:[DaemonCheck:crsd]stop {
31 2013-01-04 13:45:03.941: [ USRTHRD][1123928384] {0:6:99} Thread:[DaemonCheck:crsd]stop }
0 2013-01-04 13:45:04.943: [ora.crsd][1123928384] {0:6:99} [start] without returnbuf
2013-01-04 13:45:04.947: [ora.crsd][1123928384] {0:6:99} [start] clsdmc_respget return:
31 2013-01-04 13:45:04.947: [ora.crsd][1123928384] {0:6:99} [start] getOracleHomeAttrib: or
1 2013-01-04 13:45:04.948: [ora.crsd][1123928384] {0:6:99} [start] PID 19304 from /data/or
31 2013-01-04 13:45:04.948: [ora.crsd][1123928384] {0:6:99} [start] }DaemonAgent::start
2 2013-01-04 13:45:04.948: [ora.crsd][1123928384] {0:6:99} [start] (:CLSN00107:) clsn_agen
2013-01-04 13:45:04.948: [ AGFW][1123928384] {0:6:99} Command: start for resource: or
31 2013-01-04 13:45:04.948: [ AGFW][1111320896] {0:6:99} Agent sending reply for: RESOUR
3 2013-01-04 13:45:04.949: [ora.crsd][1123928384] {0:6:99} [check] clsdmc_respget return:
31 2013-01-04 13:45:04.950: [ USRTHRD][1123928384] {0:6:99} Thread:[DaemonCheck:crsd]start
4 2013-01-04 13:45:04.950: [ USRTHRD][1123928384] {0:6:99} Thread:[DaemonCheck:crsd]start
31 2013-01-04 13:45:04.950: [ora.crsd][1123928384] {0:6:99} [check] DaemonAgent::check retu
2013-01-04 13:45:04.951: [ AGFW][1111320896] {0:6:99} ora.crsd 1 1 state changed from
5 2013-01-04 13:45:04.951: [ AGFW][1111320896] {0:6:99} Started implicit monitor for [o
31 2013-01-04 13:45:04.951: [ AGFW][1111320896] {0:6:99} Agent sending last reply for: R
6 2013-01-04 13:45:05.757: [ USRTHRD][1109219648] {0:0:113} use all detected INF
31 2013-01-04 13:45:05.952: [ USRTHRD][1107118400] {0:6:99} Thread:[DaemonCheck:crsd]Daemon
2013-01-04 13:45:05.952: [ USRTHRD][1107118400] {0:6:99} Thread:[DaemonCheck:crsd]Initia
7 2013-01-04 13:45:05.952: [ USRTHRD][1107118400] {0:6:99} Check action requested by agent
31 2013-01-04 13:45:05.952: [ USRTHRD][1107118400] {0:6:99} Thread:[DaemonCheck:crsd]isRunn
8 2013-01-04 13:45:05.953: [ AGFW][1111320896] {0:6:99} Agent received the message: RES
31 2013-01-04 13:45:05.953: [ AGFW][1111320896] {0:6:99} Preparing CHECK command for: or
9 2013-01-04 13:45:05.953: [ COMMCRS][1123928384]clscsendx: (0x2aaab031c830) Physical conn
32
[ clsdmc][1123928384]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
0 2013-01-04 13:45:05.954: [ora.crsd][1123928384] {0:6:99} [check] Error = error 11 encoun
32 2013-01-04 13:45:05.954: [ora.crsd][1123928384] {0:6:99} [check] Calling PID check for d
1 2013-01-04 13:45:05.954: [ora.crsd][1123928384] {0:6:99} [check] Process id 19304 transl
32 2013-01-04 13:45:05.954: [ COMMCRS][1123928384]clscsendx: (0x2aaab031c830) Connection no
2
32 [ clsdmc][1123928384]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)
2013-01-04 13:45:05.954: [ora.crsd][1123928384] {0:6:99} [check] Error = error 6 encount
3 2013-01-04 13:45:05.954: [ora.crsd][1123928384] {0:6:99} [check] DaemonAgent::check retu
32 2013-01-04 13:45:05.954: [ AGFW][1111320896] {0:6:99} ora.crsd 1 1 state changed from
4 2013-01-04 13:45:05.954: [ AGFW][1111320896] {0:6:100} Generating new Tint for unplan
32 2013-01-04 13:45:05.954: [ AGFW][1111320896] {0:6:100} Agent sending message to PE: R
2013-01-04 13:45:07.752: [ USRTHRD][1109219648] {0:0:113} use all detected INF
5 2013-01-04 13:45:09.756: [ USRTHRD][1109219648] {0:0:113} use all detected INF
32 2013-01-04 13:45:11.749: [ USRTHRD][1109219648] {0:0:113} use all detected INF
6 2013-01-04 13:45:13.762: [ USRTHRD][1109219648] {0:0:113} use all detected INF
32 2013-01-04 13:45:15.707: [ AGFW][1782259808] Recvd request to shed the threads
7 2013-01-04 13:45:15.707: [CLSFRAME][1782259808] TM [MultiThread] is changing desired thr
2013-01-04 13:45:15.708: [CLSFRAME][1113422144] {0:6:102} Worker thread is exiting in TM
32 2013-01-04 13:45:15.758: [ USRTHRD][1109219648] {0:0:113} use all detected INF
8 2013-01-04 13:45:17.753: [ USRTHRD][1109219648] {0:0:113} use all detected INF
32 2013-01-04 13:45:19.757: [ USRTHRD][1109219648] {0:0:113} use all detected INF
9 2013-01-04 13:45:21.751: [ USRTHRD][1109219648] {0:0:113} use all detected INF
2013-01-04 13:45:23.756: [ USRTHRD][1109219648] {0:0:113} use all detected INF
33 [ clsdmc][1123928384]CLSDMC.C returnbuflen=8, extraDataBuf=EE, returnbuf=12FF8660
0 2013-01-04 13:45:23.882: [ora.ctssd][1123928384] {0:0:2} [check] clsdmc_respget return:
33 2013-01-04 13:45:23.882: [ora.ctssd][1123928384] {0:0:2} [check] translateReturnCodes, r
1 2013-01-04 13:45:25.762: [ USRTHRD][1109219648] {0:0:113} use all detected INF
33 2013-01-04 13:45:27.026: [ora.crf][1126029632] {0:0:2} [check] clsdmc_respget return: st
2013-01-04 13:45:27.026: [ora.crf][1126029632] {0:0:2} [check] Check return = 0, state d
2 2013-01-04 13:45:27.767: [ USRTHRD][1109219648] {0:0:113} use all detected INF
33 2013-01-04 13:45:29.771: [ USRTHRD][1109219648] {0:0:113} use all detected INF
3 2013-01-04 13:45:31.775: [ USRTHRD][1109219648] {0:0:113} use all detected INF
2013-01-04 13:45:33.780: [ USRTHRD][1109219648] {0:0:113} use all detected INF
33 2013-01-04 13:45:35.775: [ USRTHRD][1109219648] {0:0:113} use all detected INF
4 2013-01-04 13:45:37.780: [ USRTHRD][1109219648] {0:0:113} use all detected INF
33 2013-01-04 13:45:39.774: [ USRTHRD][1109219648] {0:0:113} use all detected INF
5 2013-01-04 13:45:41.778: [ USRTHRD][1109219648] {0:0:113} use all detected INF
2013-01-04 13:45:43.691: [ AGFW][1126029632] {0:0:113} Created alert : (:CRSAGF00113:
33 2013-01-04 13:45:43.691: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] (
6 2013-01-04 13:45:43.691: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] a
33 2013-01-04 13:45:43.691: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] a
7 2013-01-04 13:45:43.691: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] t
33 2013-01-04 13:45:43.693: [ USRTHRD][1104529728] {0:0:113} Thread:[NetHAMain]stop {
2013-01-04 13:45:43.790: [ USRTHRD][1109219648] {0:0:113} use all detected INF
8 2013-01-04 13:45:43.792: [ USRTHRD][1109219648] {0:0:113} [NetHAMain] thread stopping
33 2013-01-04 13:45:43.792: [ USRTHRD][1109219648] {0:0:113} Thread:[NetHAMain]isRunning is
9 2013-01-04 13:45:43.792: [ USRTHRD][1104529728] {0:0:113} Thread:[NetHAMain]stop }
34 2013-01-04 13:45:43.792: [ USRTHRD][1104529728] {0:0:113} thread cleaning up
2013-01-04 13:45:44.003: [ora.cluster_interconnect.haip][1104529728] {0:0:113} [start] S
0 2013-01-04 13:45:44.003: [ AGENT][1104529728] {0:0:113} UserErrorException: Locale is
34 2013-01-04 13:45:44.003: [ora.cluster_interconnect.haip][1104529728] {0:0:113} [start] c
1 CRS-5017: The resource action "ora.cluster_interconnect.haip start" encountered the foll
34 Start action for HAIP aborted. For details refer to "(:CLSN00107:)" in "/data/oracle/grid
2
34 2013-01-04 13:45:44.003: [ AGFW][1104529728] {0:0:113} sending status msg [CRS-5017:
Start action for HAIP aborted. For details refer to "(:CLSN00107:)" in "/data/oracle/grid
3 ] for start for resource: ora.cluster_interconnect.haip 1 1
34 2013-01-04 13:45:44.003: [ora.cluster_interconnect.haip][1104529728] {0:0:113} [start] (
4 2013-01-04 13:45:44.004: [ AGFW][1111320896] {0:0:113} Agent sending reply for: RESOU
34 2013-01-04 13:45:47.695: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] g
5 2013-01-04 13:45:47.695: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] t
2013-01-04 13:45:47.695: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] a
34 2013-01-04 13:45:47.695: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [start] (
6 2013-01-04 13:45:47.695: [ AGFW][1126029632] {0:0:113} Command: start for resource: o
34 2013-01-04 13:45:47.696: [ AGFW][1111320896] {0:0:113} Agent sending reply for: RESOU
7 2013-01-04 13:45:47.696: [ AGFW][1111320896] {0:0:113} ora.cluster_interconnect.haip
2013-01-04 13:45:47.696: [ AGFW][1111320896] {0:0:113} Agent sending last reply for:
34 2013-01-04 13:45:47.701: [ AGFW][1111320896] {0:0:113} Agent received the message: RE
8 2013-01-04 13:45:47.701: [ AGFW][1111320896] {0:0:113} Preparing CLEAN command for: o
34 2013-01-04 13:45:47.701: [ AGFW][1111320896] {0:0:113} ora.cluster_interconnect.haip
9 2013-01-04 13:45:47.702: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [clean] (
35 2013-01-04 13:45:47.703: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [clean] c
2013-01-04 13:45:47.703: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [clean] c
0 2013-01-04 13:45:47.703: [ora.cluster_interconnect.haip][1126029632] {0:0:113} [clean] (
35 2013-01-04 13:45:47.703: [ AGFW][1126029632] {0:0:113} Command: clean for resource: o
1 2013-01-04 13:45:47.703: [ AGFW][1111320896] {0:0:113} Agent sending reply for: RESOU
35 2013-01-04 13:45:47.703: [ AGFW][1111320896] {0:0:113} ora.cluster_interconnect.haip
2013-01-04 13:45:47.704: [ AGFW][1111320896] {0:0:113} Agent sending last reply for:
2 [ clsdmc][1123928384]CLSDMC.C returnbuflen=8, extraDataBuf=EE, returnbuf=12CA00C0
35 2013-01-04 13:45:53.888: [ora.ctssd][1123928384] {0:0:2} [check] clsdmc_respget return:
3 2013-01-04 13:45:53.888: [ora.ctssd][1123928384] {0:0:2} [check] translateReturnCodes, r
35 2013-01-04 13:45:57.032: [ora.crf][1123928384] {0:0:2} [check] clsdmc_respget return: st
4 2013-01-04 13:45:57.032: [ora.crf][1123928384] {0:0:2} [check] Check return = 0, state d
2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] Arg Value = chec
35 2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] getOracleHomeAtt
5 2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] Utils::getCrsHom
35 2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] Utils::execCmd 1
6 2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] Utils::getCrsHom
2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] Adding Environme
35 2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] Adding Environme
7 2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] Utils:execCmd ac
35 2013-01-04 13:45:59.825: [ora.drivers.acfs][1123928384] {0:0:2} [check] getOracleHomeAtt
8 2013-01-04 13:45:59.928: [ora.drivers.acfs][1123928384] {0:0:2} [check] execCmd ret = 0
[ clsdmc][1123928384]CLSDMC.C returnbuflen=8, extraDataBuf=EE, returnbuf=12B1DF60
35 2013-01-04 13:46:23.896: [ora.ctssd][1123928384] {0:0:2} [check] clsdmc_respget return:
9 2013-01-04 13:46:23.896: [ora.ctssd][1123928384] {0:0:2} [check] translateReturnCodes, r
36 2013-01-04 13:46:25.857: [ AGFW][1111320896] {0:0:2} Agent received the message: AGEN
0 2013-01-04 13:46:27.040: [ora.crf][1104529728] {0:0:2} [check] clsdmc_respget return: st
36 2013-01-04 13:46:27.040: [ora.crf][1104529728] {0:0:2} [check] Check return = 0, state d
[ clsdmc][1104529728]CLSDMC.C returnbuflen=8, extraDataBuf=EE, returnbuf=12B8A800
1
36
2
36
3
36
4
36
5
36
6
36
7
36
8
36
9
37
0
37
1
37
2
37
3
37
4
37
5
37
6
37
7
37
8
37
9
38
0
38
1
38
2
38
3
38
4
38
5
38
6
38
7
38
8
38
9
39
0
39
1
39
2
39
3
39
4
39
5
39
6
39
7
39
8
39
9
40
0
40
1
40
2
40
3
40
4
40
5
40
6
40
7
40
8
40
9
41
0
41
1
41
2
41
3
41
4
41
5
41
6
41
7
41
8
41
9
42
0
42
1
42
2
42
3
42
4
42
5
42
6
42
7
42
8
42
9
43
0
43
1
43
2
43
3
43
4
43
5
43
6
43
7
43
8
43
9
44
0
44
1
44
2
44
3
44
4
44
5
44
6
44
7
44
8
44
9
45
0
45
1
45
2
45
3
45
4
45
5
45
6
45
7
45
8
45
9
46
0
46
1
46
2
46
3
46
4
46
5
46
6
46
7
46
8
46
9
47
0
47
1
47
2
47
3
47
4
47
5
47
6
47
7
47
8
47
9
48
0
48
1
48
2
48
3
48
4
48
5
48
6
48
7
48
8
48
9
49
0
49
1
49
2
49
3
49
4
49
5
49
6
49
7
49
8
49
9
50
0
50
1
50
2
50
3
50
4
50
5
50
6
50
7
50
8
50
9
51
0
51
1
51
2
51
3
51
4
51
5
51
6
51
7
51
8
51
9
52
0
52
1
52
2
52
3
52
4
52
5
52
6
52
7
52
8
52
9
53
0
53
1
53
2
53
3
53
4
53
5
53
6
53
7
53
8
53
9
54
0
54
1
54
2
54
3
54
4
54
5
54
6
54
7
54
8
54
9
55
0
55
1
55
2
55
3
55
4
55
5
55
6
55
7
55
8
55
9
56
0
56
1
56
2
56
3
56
4
56
5
56
6
56
7
56
8
56
9
57
0
57
1
57
2
57
3
57
4
57
5
57
6
57
7
57
8
57
9
58
0
58
1
58
2
58
3
58
4
58
5
58
6
58
7
58
8
58
9
59
0
59
1
59
2
59
3
59
4
59
5
59
6
59
7
59
8
59
9
60
0
60
1
60
2
60
3
60
4
60
5
60
6
60
7
60
8
60
9
61
0
61
1
61
2
61
3
61
4
61
5
61
6
61
7
61
8
61
9
62
0
62
1
62
2
62
3
62
4
62
5
62
6
62
7
62
8
62
9
63
0
63
1
63
2
63
3
63
4
63
5
63
6
63
7
63
8
63
9
64
0
64
1
64
2
64
3
64
4
64
5
64
6
64
7
64
8
64
9
65
0
65
1
65
2
65
3
65
4
65
5
65
6
65
7
65
8
65
9
66
0
66
1
66
2
66
3
66
4
66
5
66
6
66
7
66
8
66
9
67
0
67
1
67
2
67
3
67
4
67
5
67
6
67
7
67
8
67
9
68
0
68
1
68
2
68
3
68
4
68
5
68
6
68
7
68
8
68
9
69
0
69
1
69
2
69
3
69
4
69
5
69
6
69
7
69
8
69
9
70
0
70
1
70
2
70
3
70
4
70
5
70
6
70
7
70
8
70
9
71
0
71
1
71
2
71
3
71
4
71
5

There is a lot of log messages but for me none of them is very helpful. Let's get more detail about what
exactly is running and what is not running:

?
1 [root@vol57-1 ~]# crsctl stat res -t -init
--------------------------------------------------------------------------------
2 NAME TARGET STATE SERVER STATE_DETAILS
3 --------------------------------------------------------------------------------
4 Cluster Resources
5 --------------------------------------------------------------------------------
ora.asm
6 1 ONLINE ONLINE vol57-1 Started
7 ora.cluster_interconnect.haip
8 1 ONLINE OFFLINE
9 ora.crf
10 1 ONLINE ONLINE vol57-1
ora.crsd
11 1 ONLINE OFFLINE
12 ora.cssd
13 1 ONLINE ONLINE vol57-1
14 ora.cssdmonitor
1 ONLINE ONLINE vol57-1
15 ora.ctssd
16 1 ONLINE ONLINE vol57-1 OBSERVER
17 ora.diskmon
18 1 OFFLINE OFFLINE
ora.drivers.acfs
19 1 ONLINE ONLINE vol57-1
20 ora.evmd
21 1 ONLINE INTERMEDIATE vol57-1
22 ora.gipcd
23 1 ONLINE ONLINE vol57-1
ora.gpnpd
24 1 ONLINE ONLINE vol57-1
25 ora.mdnsd
26 1 ONLINE ONLINE vol57-1
27
28
29
30
31
32

Normally they are all ONLINE, now ora.cluster_interconnect.haip, ora.crsd, ora.diskmon and ora.evmd
are not ONLINE. Nice to see that ora.asm is ONLINE. crsd had been started with the start cluster
command and now is down again. check the log for it. csrd.log:

expand source

It clearly shows there is a problem with the network setup which is a bit strange, the cluster has been
running for about a year, before I upgraded it from 11.2.0.3.2 to 11.2.0.3.4. The upgrade did not run
the smoothest but eventually succeeded and I used it for several tests. Networking is also not exactly
the piece of cake I eat as favorite but it looks like there is no escaping here.

Oracle maintains a GPnP profile to make life for dba's easier, without the need for a network admin.
To get the GPnP profile:

?
1[root@vol57-1 ~]# gpnptool get
2Warning: some command line parameters were defaulted. Resulting command line:
3 /data/oracle/grid/11.2.0.3.0/bin/gpnptool.bin get -o-
4
5<!--?xml version="1.0" encoding="UTF-8"?-->
61XXiTvVH4DTEwnzsthvxVJLC6a8=Ogi762u9e9zSPwQL7c7GxBIdgt3mdCBSAuV8k3i+ET3dxjOPyLwazrJ5nPGyJ
Success.
7

Edited for better reading:

?
1 <?xml version="1.0" encoding="UTF-8"?>
2 <gpnp:GPnP-Profile xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http:
3 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Version="1.0" xsi:schemaLocation="h
ClusterName="vol57-cluster" PALocation="">
4 <gpnp:Network-Profile>
5 <gpnp:HostNetwork id="gen" HostName="*">
6 <gpnp:Network id="net1" IP="192.168.2.0" Adapter="eth0" Use="public"/>
7 <gpnp:Network id="net2" IP="192.168.100.0" Adapter="eth1" Use="cluster_interco
</gpnp:HostNetwork>
8 </gpnp:Network-Profile>
9 <orcl:CSS-Profile id="css" DiscoveryString="+asm" LeaseDuration="400"/>
10 <orcl:ASM-Profile id="asm" DiscoveryString="/dev/oracle/*" SPFile="+DATA/vol57-cluster/
11 <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
12 <ds:SignedInfo>
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"
13 <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
14 <ds:Reference URI="">
15 <ds:Transforms>
16 <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-s
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
17 <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14
18 </ds:Transform>
19 </ds:Transforms>
20 <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
21 <ds:DigestValue>1XXiTvVH4DTEwnzsthvxVJLC6a8=</ds:DigestValue>
</ds:Reference>
22 </ds:SignedInfo>
23 <ds:SignatureValue>Ogi762u9e9zSPwQL7c7GxBIdgt3mdCBSAuV8k3i+ET3dxjOPyLwazrJ5nPGyJQG
24 </ds:Signature>
25</gpnp:GPnP-Profile>
26
27
28
29

It shows amongst others that the eth0 and eth1 interfaces play a role in this cluster. Next to that it also
shows the location of the spfile and the ASM discovery string. The files can be found in
$ORACLE_HOME/gpnp/. The profile has not been changed since the cluster was created.

?
1
[root@vol57-1 ~]# ifconfig eth1
2 eth1 Link encap:Ethernet HWaddr 00:0C:29:2D:D0:EF
3 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
4 RX packets:16223 errors:0 dropped:0 overruns:0 frame:0
5 TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
6 RX bytes:7565637 (7.2 MiB) TX bytes:42 (42.0 b)
7

Let's give it a manual IP address.

?
1 [root@vol57-1 ~]# ifconfig eth1 inet 192.168.100.1
and see if we can start the ora.cluster_interconnect.haip when the address is configured.

?
1 [root@vol57-1 ~]# crsctl start res ora.cluster_interconnect.haip -init
2 CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vol57-1'
3 CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'vol57-1' succeeded

That is better. On vol57-2

?
1
2 [root@vol57-2 ~]# ifconfig eth1
eth1 Link encap:Ethernet HWaddr 00:1C:42:7F:C6:A8
3 inet addr:192.168.100.2 Bcast:192.168.100.255 Mask:255.255.255.0
4 inet6 addr: fe80::21c:42ff:fe7f:c6a8/64 Scope:Link
5 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
6 RX packets:17195 errors:0 dropped:0 overruns:0 frame:0
TX packets:18597 errors:0 dropped:0 overruns:0 carrier:0
7 collisions:0 txqueuelen:1000
8 RX bytes:3712399 (3.5 MiB) TX bytes:17535373 (16.7 MiB)
9

So these nodes can talk to each other. With a little luck the other processes can also be started now.
crsd depends on the interconnect so it might work:

?
1 [root@vol57-1 gpnp]# crsctl start res ora.crsd -init
2 CRS-2672: Attempting to start 'ora.crsd' on 'vol57-1'
3 CRS-2676: Start of 'ora.crsd' on 'vol57-1' succeeded

Since now crs is running, we should be able to use the regular commands, without the "-init" flag.

expand source

Nice to see this happen. One node is up and running, the databases are starting on vol57-1, now fix
node 2, vol57-2.

?
1 [root@vol57-2 ~]# crsctl start res ora.cssd -init
CRS-2672: Attempting to start 'ora.cssd' on 'vol57-2'
2 CRS-2672: Attempting to start 'ora.diskmon' on 'vol57-2'
3 CRS-2676: Start of 'ora.diskmon' on 'vol57-2' succeeded
4 CRS-2676: Start of 'ora.cssd' on 'vol57-2' succeeded
5 CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'vol57-2'
CRS-2672: Attempting to start 'ora.ctssd' on 'vol57-2'
6 CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'vol57-2' succeeded
7 CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vol57-2'
8 CRS-2676: Start of 'ora.ctssd' on 'vol57-2' succeeded
9 CRS-2672: Attempting to start 'ora.evmd' on 'vol57-2'
10 CRS-2676: Start of 'ora.evmd' on 'vol57-2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'vol57-2' succeeded
11 CRS-2679: Attempting to clean 'ora.asm' on 'vol57-2'
12 CRS-2681: Clean of 'ora.asm' on 'vol57-2' succeeded
13 CRS-2672: Attempting to start 'ora.asm' on 'vol57-2'
14 CRS-2676: Start of 'ora.asm' on 'vol57-2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'vol57-2'
15 CRS-2676: Start of 'ora.crsd' on 'vol57-2' succeeded
16
17
18
19
20

Now the cluster services are running and we are back online with the full cluster like it used to be.
Question that remains is: how was this caused? I think I handled the system a bit rude when I resetted
one of the vm hosts, otherwise there have been no changes. For now, I am happy with it, after a restart
everything came back online as normal. The root cause of the problem appeared to be that some
other device was using the designated address for the interconnect. A bit strange since I like to think
this net is not in used in my lab.

?
1 [root@vol57-1 ~]# ifup eth1
2 Error, some other host already uses address 192.168.100.1.