LinuxTroubleshootingCheatsheet:strace,htop,lsof,tcpdump,iftop&

sysdig

Thischeatsheetisagreatguideofcommandlineslinuxadminscanusetogetinsightsintotheirservers.Whetheryouvebeenanadminforonemonthor20years
youvedefinitelyusedoneifnotallofthesetoolstotroubleshootanissue.Becausewelovesysdig(naturally!)wealsoincludedatranslationforeachofthese
commonoperationsintothesysdigcommandlineorcsysdig.
Ratherthanattemptcoveringalloptionsfrommanpages(whichwouldhavemadeforboringcoverageofmanyesoteric,rarelyusedswitches),wevestartedfrom
examplesreferencedatthemostpopularwebpagesyoudfindwhenyousearchfortermslikestraceexamples,htopexamples,andsoforth.
Doyouhavefavoritesthatarentlistedhere?Letusknowandwellincludetheminfuturearticles.

strace

Theresonesubtledifferencebetweenstraceandsysdigthatwillbeapparentinmanyofthesesidebysidecomparisons:Manyofthesimpleststraceexamples
includecommandlinesthatareexecutedandtracedasaoneshotoperation.Ontheotherhand,sysdighasasomewhatdifferentphilosophy,inthatiteither
watchesliveeventsfromafarastheyhappen,oranalyzescapturedatapreviouslysavedtoafile.Thankfully,sysdig'srichfilteringoptionsprovidetheknobstowatch
forspecificoneshotexecutions,asyoullsoonsee.

Operation

strace

sysdig

Note

Tracetheexecutionofa
command

stracewho

sysdigproc.name=who

Whereasstracerunsthe
who
command
shownhereasaoneshot,sysdigis
watchingfortheexecutionofw

ho
.Use
sysdigsfilteringtofurtherisolatea
specificrun,e.g.:

sysdigproc.name=whoandproc.ppid=534

Thiswatchesfora
who
thatsabouttobe
runinashellthatyouvedeterminedto
havePIDof534.
Traceonlywhencertain
specificsystemcallsare
made

straceeopenwho
sysdigevt.type=openandproc.name=who

straceetrace=open,readwho sysdig"evt.typein(open,read)andproc.name=who"

Saveatracetoafile

straceooutput.txtwho

sysdigwoutput.scapproc.name=who

Withstrace,thefileproducedcontains
thesametextyoudhaveviewedonthe
screenifruninteractively.Withsysdig,
yougetaraw,reusablecapturefile,
suchthatyoucanviewthetextoutput
with:

sysdigroutput.scap

Youcouldalsousethisasthebasisto
applyfiltersoranyothersysdig
functionalityyouwanttoapplyasyou
revisittheoriginalevents.
Watcharunningprocess
withPID=1363

stracep1363

sysdigproc.pid=1363

Printatimestampfor
eachoutputlineofthe
trace

stracetwho

sysdigproc.name=who

Sysdigprintstimestampsbydefault.

Printrelativetimefor
systemcalls

stracerwho

sysdigtDproc.name=who

Sysdigoffersseveralmorewaysto
representtimestampsviathe

t
option.

Generatebatchstatistics
reportsofsystemcalls

stracecwho

sysdigwoutput.scapproc.name=who
#Nowrunthewhoseparately

Sysdigsdefaultbehaviorismore
optimizedforthecaseofpresenting
eventdataasithappensratherthan
batchreporting.Thisiswhythesysdig
equivalentisdoneintwostepshere.

Foroneshotbatchtextreports:
sysdigroutput.scapctopscallsc
topscalls_time

Orforaninteractivereportthatallowsforfurther
drilldown:
csysdigroutput.scapvsyscalls

Generatelive,persecond N/A
statisticsreportsof
systemcallsforrunning
processwithPID=1363

csysdigvsyscallsproc.pid=1363

Whilestracecanshowindividualevents
astheyhappenlive,orprovideasingle
batchreportfortheexecutionofa
command,csysdigsviewsprovidea
uniqueabilitytoshowlive,periodic
reports.


htop

Sincehtopisalive,interactive,cursesstyletool,wellcompareittothelive,interactive,cursesstylecsysdig.

Forstarters,bothtoolsusethesameapproachofnavigatingthelivetableviaUp/Down/Left/RightarrowsandalsoPgUp/PgDn.Foroperationsthataffectasingle
process(killing,renicing,etc.)itisassumedyouveusedthesecontrolstofirsthighlightaparticularprocess.

Operation

htop

csysdig

Note

Changesortorderbased
onacolumnofthetable

Press
F6
,
<
,or
>
andthenselectacolumnby
name,or

PressM

,P

,orT

tosortbyMemory,

ProcessorUsage,orTime

PressI

toinvertthesortorder

Press
F9
or
>
andthenselectacolumnby
name,or

Press<

shift><19>
tosortbyany
column<

n>
,andpressrepeatedlyto
invertsortorder,or

Mouseclickonacolumnheader

Killaprocess

Press
F9
or
k

Press
k

Reniceaprocess

Press
F7
or
]
toreducethenicevalueby1
Press
F8
or
[
toincreasethenicevalueby1

Press
]
toreducethenicevalueby1
Press
[
toincreasethenicevalueby1

Thisillustrateshoweasyitistocustomize
sysdig.Inoticedwhenfirstwritingthis
articlethatcsysdigwasmissingacouple
minorfeatureslikethis,soIusedthe
opportunitytolearnhoweasyitisto
write/modifyChisels,thenputupmy
improvementsasa
PullRequest
.Youcan
dothesame!

Displayonlyprocesses
startedbyausernamed
"phil"

Press
u
,then
Selecttheusernamep

hil
fromthelist

Launchas:

Changetheoutput
refreshintervaltoonce
every5seconds

Launchas:

Launchas:

htopd50

csysdigd5000

csysdiguser.name=phil

Ormouseclick
Filter:
fromwithin
csysdig
atthetopofthedefault
Processesview,thenappend
and
user.name=phil
tothecurrentfiltertext
Asyoucansee,htopworksinunitsof
tenthsofasecond,whilecsysdigworksin
milliseconds.

Startasystemcalltrace
onaprocess

Press
s
tostartan
strace

Press
F6
tostarta
sysdig

Listopenfilesfora
process

Press
l
torunaonetime
lsof

Press
f
torunaonetimelsof

Ortoseerealtime,updatingreportsof
files/directoriesusedaprocess,drill
downtoaspecificprocessbypressing
Enter
,thenpressF

2
andselectaView
suchasF

iles
,F

ileOpensList
,or
Directories
.

SeetheNoteaboveforReniceaprocess
abouthowtheonetimel

sof
wasrecently
addedasanenhancement.

Followaprocess,such
thatitremains
highlightedevenasits
orderinthelistchanges

Press
F

Defaultbehavioristoalwaysfollowthe
highlightedprocess

lsof

Operation

lsof

sysdig

Note

Listallopenfilesbelongingto
allactiveprocesses

lsof

sysdigclsof

Listprocessesthathave
openedthespecificfile
/var/log/syslog

lsof/var/log/syslog

sysdigclsof"fd.name=/var/log/syslog"

Listprocessesthathave
openedfilesunderthe
directory/var/log

lsof+d/var/log

sysdigclsof"fd.directory=/var/log"

Listfilesopenedbyprocesses
namedsshd

lsofcsshd

sysdigclsof"proc.name=sshd"

Listfilesopenedbyaspecific
usernamedphil

lsofuphil

sysdigclsof"user.name=phil"

Listfilesopenedbyeveryone
exceptfor
theusernamed

lsofu^phil

sysdigclsof"user.name!=phil"

phil
Listallopenfilesforaspecific
processwithPID=1081

lsofp1081

sysdigclsof"proc.pid=1081"

Listallfilesopenedbyuser
"phil"oraprocessnamed
"sshd"(ORlogic)

lsofuphilcsshd

sysdigclsof"'user.name=philor
proc.name=sshd'"

Notetheuseoftwolayersofquoteswith
thesysdigfilter.

Listallfilesopenedbyan
"sshd"processforuser"phil"
(ANDlogic)

lsofuphilcsshda

sysdigclsof"'user.name=philand
proc.name=sshd'"

Notetheuseoftwolayersofquoteswith
thesysdigfilter.

Observerepeatingreportsof
openfilesbasedonlive
activity

Enablerepeatmodewithoneof: Similarlivedatacanbeobtainedwitha
lsofr
live/interactivecsysdigview,launchedlikeso:
lsof+r

csysdigvfiles
csysdigvfile_opens

Listallnetworkconnections

lsofi

sysdigclsof"fd.type=ipv4"

Listnetworkconnectionsin
usebyaspecificprocesswith
PID=1014

lsofiap1014

sysdigclsof"'fd.type=ipv4and
proc.pid=1014'"

Notetheuseoftwolayersofquoteswith
thesysdigfilter.

Listprocessesthatare
listeningonport22

lsofi:22

sysdigclsof"'fd.port=22and
fd.is_server=true'"

Notetheuseoftwolayersofquoteswith
thesysdigfilter.

ListallTCPorUDP
connections

lsofitcp

lsofiudp

sysdigclsof"fd.l4proto=tcp"

sysdigclsof"fd.l4proto=udp"

tcpdump

tcpdumpisfocusedentirelyonnetworktraffic,whilenetworktrafficisonlyasubsetofwhatSysdigcovers.Manytcpdumpusecasesinvolvefiltering,andtcpdump
usesnetworkspecific
BPFfilters
,whereasSysdigusesitsownbroaders
ysdigfiltering
.Thetwoapproacheslooksimilarinmanyways,butyoullwanttolookatthe
docsforeachsidebysideasyouprogresstomoreadvancedfilteringneeds.Also,sinceinLinux
everythingisafile
,youllnoticetheSysdigfilteringexamplesbelowall
leverageanetworkconnectionsviafiledescriptorsapproach.

Operation

tcpdump

Sysdig

Note

Capturepacketsfroma
particularinterfaceeth0
(192.168.10.119)

tcpdumpieth0

sysdigfd.ip=192.168.10.119

Sysdigdoesnotcurrentlyhavefiltering
basedonnamedinterfaces,butthe
equivalentviaIPaddressisshownhere.

Captureonly100packets

tcpdumpc100

sysdign100fd.type=ipv4

Displaycapturedpackets
inASCII

tcpdumpA

sysdigAfd.type=ipv4

Displaycapturedpackets
inHEXandASCII

tcpdumpXX

sysdigXfd.type=ipv4

Capturepacketdata,
writingitintointoafile

tcpdumpwsaved.pcap

sysdigwsaved.scapfd.type=ipv4

Thesysdigfileformatiscapableof
holdingeventdataformuchmorethan
justnetworkpackets(e.g.systemcalls).

Readbacksavedpacket
datafromafile

tcpdumprsaved.pcap

sysdigrsaved.scap

Captureonlypackets
longer/smallerthan1024
bytes

tcpdumpgreater1024

tcpdumpless1024

sysdig"fd.type=ipv4andevt.buflen>1024"

sysdig"fd.type=ipv4andevt.buflen<1024"

The
greater/less
optionsintcpdump
referenceoverallpacketlengthwhereas
evt.buflen
insysdigisrelativeto
payloadsize.

CaptureonlyUDPorTCP
packets

tcpdumpudp

tcpdumptcp

sysdigfd.l4proto=udp

sysdigfd.l4proto=tcp

Notethatwedontneedtoexplicitly
includef

d.type=ipv4
sincewereusing
othernetworkonlyfiltershere.

Captureonlypackets
goingto/fromaparticular
port

tcpdumpport22

sysdigfd.port=22

Notethatwedontneedtoexplicitly
includef

d.type=ipv4
sincewereusing
othernetworkonlyfiltershere.

Capturepacketsfora
particulardestinationIP
andport

tcpdumpdst54.165.81.189
andport6666

sysdigfd.rip=54.165.81.189andfd.port=6666

Notethatwedontneedtoexplicitly
includef

d.type=ipv4
sincewereusing
othernetworkonlyfiltershere.

iftop

Sinceiftopisalive,interactive,cursesstyletool,wellcompareittothelive,interactive,cursesstylecsysdig.Also,liketcpdump,iftopuses
BPFfilters
.Seethe
previousintrotothesectionontcpdumpformoredetailaboutfilteringdifferences.

Operation

iftop

Csysdig

Note

Displayatableofcurrent
bandwidthusage
betweenpairsofhosts

iftop

Launchas:

Bydefaultiftopwatchesjustthefirst
interfaceitfinds,whereasbydefault
csysdigwatchestrafficacrossthe
entirehost.

Turnondisplayof
networkports

Launchas:

Observetrafficforjust
theeth0interface
(192.168.10.119)

ResolveDNSnames

Press
n
fromwithin
iftop
totoggle
resolutionforallhostsshown

Press
n
fromwithin
csysdig
torun
nslookup
on
thecurrentlyhighlightedremotehost

Changesortorderbased
onacolumnofthetable

Press
<
tosortbysource
Press
>
tosortbydestination

Press
F9
or
>
andthenselectacolumnbyname,
or

Press<

shift><19>
tosortbyanycolumn<

n>
,
andpressrepeatedlytoinvertsortorder,or

Mouseclickonacolumnheader

Filtertoshowonlytraffic
goingto/fromIPaddress
54.84.222.1

Launchas:

Launchas:

iftopf"host54.84.222.1"

csysdigvconnectionsfd.ip=54.84.222.1

Pausethedisplay

Press
P

Press
p

Scrollthedisplay

Press
j
toscrollup
Press
k
toscrolldown

Press
Up/Down/Left/Right
arrowsor
PgUp/PgDn
to
scrollthroughthetable

Sysdig/csysdiggowellbeyond
scrollingthroughasingletable,since
youcandrilldownintothe

csysdigvconnections

OrpressF

2
fromwithin
csysdig
tochangethe
View,thenuparrowtoselect
Connections
Defaultbehavioristoalwaysdisplayports

Launchas:

Launchas:

iftopieth0

csysdigvconnectionsfd.ip=192.168.10.119

Sysdig/csysdigdonotcurrentlyhave
filteringbasedonnamedinterfaces,
buttheequivalentviaIPaddressis
shownhere.

iftopP

Orpressp

fromwithin

iftop

Ormouseclickon
Filter:
fromwithin
csysdig
,
thenappenda

ndfd.ip=192.168.10.119
tothe
existingfiltertext

Ormouseclickon
Filter:
fromwithin
csysdig
,
thenappenda

ndfd.ip=54.84.22.1
totheexisting
filtertext

ConnectionsViewtoseedatainother
groupingssuchaspercontaineror
perthread.

Acknowledgement

Theauthorwouldliketoacknowledge
www.thegeekstuff.com
,asmostoftheexamplefilledarticlesusedforthetableabovewerefoundattheirsite.