8/12/2015

OracleTechnologyStack:ExadataCommandstogenerateLogsforTroubleshooting

Wednesday,August6,2014

ExadataCommandstogenerateLogsforTroubleshooting
1.Specifytheexacttimeofissuetofocus.
2./var/log/messagesfilecoveringthetimeofrebootfrombothnodes.
3.Opatch
cd$ORACLE_HOME/OPatch
./opatchlsinventory
4)Pleaseuploaddiagcollection.shoutputfilesfromallDBnodes.
#toknowtheevictioninformationfromclusterfront.
$GRID_HOME/bin/diagcollection.sh
FordiagcollectionrefertoCRS10gR2/11gR1/11gR2DiagnosticCollectionGuide(DocID330358.1)
#script/tmp/diag.log
#id
#env
#cd<tempdirectorywithplentyfreespace>
#$GRID_HOME/bin/diagcollection.sh
#exit
Thefollowing.gzfileswillbegeneratedinthecurrentdirectoryandneedtobeuploadedalongwith/tmp/diag.log:
crsData_<hostname>.tar.gz,
ocrData_<hostname>.tar.gz,
oraData_<hostname>.tar.gz,
os_<hostname>.tar.gz
4)Pleaseuploadalert.logfromDBandASMinstancesfromallDBnodes
5)Gridhomelogsfromalldbnodes.
data:text/htmlcharset=utf8,%3Ch2%20class%3D%22dateheader%22%20style%3D%22margin%3A%200px%3B%20position%3A%20relative%3B%20fontsize%3A%2011px%3B%20minheight%3A%200px%3

1/5

8/12/2015

OracleTechnologyStack:ExadataCommandstogenerateLogsforTroubleshooting

%cd$grid_home/log/<hostname>
%find.|xargsgrep'20111129'sl
or
cd$GRID_HOME/log
tarcvzfnode1.tar.gz*
6)Celltrace&logs
Uploadlogsformepleasefromtheproblemcell,coveringproblemtime.
%cd$ADR_BASE/diag/asm/cell/<hostname>/trace/
%egreplr20120906*|xargstarcvfz`hostnames`_cell_diag.tar.gz
7)PleaseuploadtheStorageCellalert.logandStorageCellalerthistory:fromallcellnodes
a.)/opt/oracle/cell/log/diag/asm/cell/{nodename}/trace/alert.log
b.)#cellclilshowalerthistory
8)OSwatcherinformation:
locationoflogs==>/opt/oracle.oswatcher/osw/archive
cd/opt/oracle.oswatcher/osw/archive
find.name'*12.09.05*'execzip/tmp/osw_`hostnamea`.zip{}\
cd/opt/oracle.oswatcher/osw/archive
>Changedateandtimecoveringproblemtime.
find.name"*11.10.13.1[89]00*"execzip/tmp/osw_`hostnamea`_0503.zip{}\
>pleasedonotmiss''atthelastintheabovecommand
TogetOSwatcherdataofspecificdate:
cd/opt/oracle.oswatcher/osw/archive
find.name'*12.01.13*'printexeczip/tmp/osw_`hostname`.zip{}\
where12year01Month13day

data:text/htmlcharset=utf8,%3Ch2%20class%3D%22dateheader%22%20style%3D%22margin%3A%200px%3B%20position%3A%20relative%3B%20fontsize%3A%2011px%3B%20minheight%3A%200px%3

2/5

8/12/2015

OracleTechnologyStack:ExadataCommandstogenerateLogsforTroubleshooting

9)LinuxOFA&kernelinformation
a)dclilrootc<listofdbnodes>"rpmqa|grepofa"
b).dclilrootc<listofdbnodes>"unamea"
10)Linuxcrashfileinformation
#tocheckthepossibilityofLINUXPROBLEM
greppath/etc/kdump.confwillgivethecorefilelocation
Pleasecheckforavmcore/crashcorethatmayhavebeengeneratedduringthereboot.
ThecellsanddatabaseserversofOracleExadataDatabaseMachineareconfiguredtogenerateLinuxkernelcrashcorefileswhenthereisa
Linuxcrash.
Commonlocationsare/var/crashor/u01/crashfilesondatabasenodes,and/var/log/oracle/crashfilesoncellnodes.
Startinthisfile:
cat/etc/kdump.conf
Lookfortheuncommentedfilesystem(ext3)andpath.
example:
ext3/dev/md11
path/crashfiles
Thelocationisthe/crashfilesfolderfromthemountpointof/dev/md11.
Thefindwheremd11ismounted:
df
example:
/dev/md112395452280896199287213%/var/log/oracle
Thenchangetothatdirectory:
cd/var/log/oracle/crashfiles

11)SOSreport
sosreportrequiresrootpermissionstorun.
data:text/htmlcharset=utf8,%3Ch2%20class%3D%22dateheader%22%20style%3D%22margin%3A%200px%3B%20position%3A%20relative%3B%20fontsize%3A%2011px%3B%20minheight%3A%200px%3

3/5

8/12/2015

OracleTechnologyStack:ExadataCommandstogenerateLogsforTroubleshooting

#/usr/sbin/sosreport
Thesosreportwillrunforseveralminutes,accordingtodifferentsystem,therunningtimemaybemorelonger.
Oncecompleted,sosreportwillgenerateacompressedabz2fileunder/tmp.

12)Runinfinicheck(fromanyofthedbnodes)command:
#/opt/oracle.SupportTools/ibdiagtools/infinicheckg/opt/oracle.SupportTools/onecommand/dbs_ib_group|tee
/tmp/infinicheck_`hostname`.log
wheredbs_ib_groupcontainsthehostnames/IPoIBofthecomputenodespartofthecluster.IthastobeusingtheIBsubnet.
Note:Thelocationfordbs_ib_groupisforOracleSunDBMachine.Forothersystems,specifythelocationofthefile
13)Runsundiag.sh
ThescriptneedstobeexecutedasrootontheExadataStorageserverhavingdiskproblemsandsometimesalsoonDbnodesorStorage
Serversforsomeotherhardwareissues.
Forgatheringsundiag.shoutputacrossawholerackusingdcli,theoutputsmayendupwiththesametarballnamewhichwilloverwrite
eachotheruponunzipping.Toavoidthis,usethefollowingfromDB01:
1.[root@exadb01~]#cd/opt/oracle.SupportTools/onecommand(orwherevertheall_groupfileiswiththelistoftherackhostnames)
2.[root@exadb01onecommand]#dcligall_grouplroot/opt/oracle.SupportTools/sundiag.sh2>&1
<thiswilltakeuptoabout2minutes>
3.Verifythereisoutputin/tmponeachnode:
[root@exadb01onecommand]#dcligall_grouplrootserial'lsl/tmp/sundiag*'
ForgatheringOSWatcherdataalongsidesundiag:
#/opt/oracle.SupportTools/sundiag.shosw
Executionwillcreateadatestampedtar.bz2filein/tmp/sundiag_/tar.bz2includingOSWatcherarchivelogs.Theselogsmaybevery
large.
data:text/htmlcharset=utf8,%3Ch2%20class%3D%22dateheader%22%20style%3D%22margin%3A%200px%3B%20position%3A%20relative%3B%20fontsize%3A%2011px%3B%20minheight%3A%200px%3

4/5

8/12/2015

OracleTechnologyStack:ExadataCommandstogenerateLogsforTroubleshooting

#/opt/oracle.SupportTools/sundiag.shh
OracleExadataDatabaseMachineDiagnosticsCollectionTool
Version:1.5.1_20140521
Usage:./sundiag.sh[osw][ilom|snapshot]
oswCopyExaWatcherorOSWatcherlogfiles(Canbeseveral100MB's)
ilomUserlevelILOMdatagatheringoptionviaipmitool,inplaceof
separatelyusingrootlogintogetILOMsnapshotoverthenetwork.
snapshotCollectsnodeILOMsnapshotrequireshostrootpasswordforILOM
tosendsnapshotdataoverthenetwork.
ExadataDiagnosticCollectionGuide(DocID1353073.2)

data:text/htmlcharset=utf8,%3Ch2%20class%3D%22dateheader%22%20style%3D%22margin%3A%200px%3B%20position%3A%20relative%3B%20fontsize%3A%2011px%3B%20minheight%3A%200px%3

5/5