Logo ref: http://sourceware.

org/systemtap/wiki/LW2008SystemTapTutorial
SystemTap Tutorial - Part 1
Who is doing maimum read/write on my ser!er"
#an $ add some de%ug statements in the kernel without re%uilding& re%ooting the
system"
'ou might ha!e asked these (uestion to yourself& if you are a System )dministrator or a
*ernel +e!eloper.
Lets see what are our choices to answer a%o!e (uestions:,
Tracing , -ro!ides info while running and gi!es (uick o!er!iew of code flow %ut gi!es
lot of information. Tools like strace, ltrace and ftrace are used for tracing.
Profiling , $t does the sampling while running and we can do the analysis after the
e!ent has occurred. Oprofile is used for sampling.
Debugging , We can set %reakpoints& look at the !aria%les& memory& registers stack
trace etc.We can de%ug only one program at a time and de%ugger stops it while we do
the inspection. GDB/KDB is used for such de%ugging.
So& which of the a%o!e mentioned tool you will use. 'ou might %e thinking of using
com%ination of a%o!e mentioned tools. Won.t it %e great to ha!e all the capa%ilities from
a%o!e tools in one tool"
Welcome to SystemTap
SystemTap can monitor system wide multiple synchronous and asynchronous e!ents at
the same time. $t can do scripta%le filtering and statistics collection. $ts a dynamic
method of monitoring and tracing the operations of a running Linu kernel.
To instrument the running kernel SystemTap uses Kprobes and return probes.
With kernel de%ug information it gets the addresses for functions and !aria%les
referenced in the script. With utrace systemtap supports pro%ing user,space eecuta%les
and shared li%raries as well.
SystemTap is useful to System )dministrators& *ernel +e!elopers& Support /ngineers&
0esearchers and students.
!nstallation
1n 2edora
yum install systemtap kernel,de!el
yum ,,ena%lerepo3fedora,de%uginfo ,,ena%lerepo3updates,de%uginfo install

Similarly to use SystemTap on 4%untu or any other distro you need to install the
systemtap package and corresponding kernel.s de%uginfo packages.
'ou need to %e a root user to run the SystemTap scripts. 'ou can also add a normal user
to either 5stap"e#5 or 5stapusr5 group to allow him/her to run the script.

$o% it %or&s'
To understand it let run a script in !er%ose mode 6,!7.
8 stap ,! ,e .pro%e syscall.read 9printf65syscall :s arguments :s ;n5& name& argstr7<
eit67=.
-ass >: parsed user script and ?@ li%rary script6s7 using 8A@B?!irt/20C28res/2C>2shr k%&
in >@0usr/>0sys/2CBreal ms.
-ass 2: analyDed script: > pro%e6s7& C function6s7& 0 em%ed6s7& 0 glo%al6s7 using
2>?2?0!irt/>>@??0res/EAB?Cshr k%& in @?0usr/20sys/BC?real ms.
-ass A: translated to # into
5/tmp/stap4FGeHi/stapI%C0c82?8c8Eacc?8AfE@ded?2a@2ee??I2>>A.c5 using
2>?2?0!irt/>>E>80res/E@C8Cshr k%& in A20usr/C0sys/>0>Creal ms.
-ass C: compiled # into 5stapI%C0c82?8c8Eacc?8AfE@ded?2a@2ee??I2>>A.ko5 in
A0>0usr/>2>0sys/>28>8real ms.
-ass @: starting run.
syscall rea" arguments (, )*))))+fffa++,b(c), -1./
-ass @: run completed in 20usr/?0sys/>ECreal ms.
The stap program is the front-en" to the SystemTap tool. .-e0 tells it to run a script
from the net argument1
-ass >,2
-arse the script and the code is checked for semantic and synta errors.
)ny tapset reference is imported. +e%ug data pro!ided !ia de%uginfo packages
are read to find addresses for functions and !aria%les referenced in the script.
-ass A
Translate the script into # code.
-ass C
#ompile the translated # code and create a kernel module.
-ass @
$nsert the module in the kernel.
1nce the module is loaded& pro%es are inserted at proper locations. 2rom now on
whene!er a pro%e is hit& handler for that pro%e is called.
Synta to write an e!ent and its handler.
probe 2e#ent3 4 5an"ler 6
Where e#ent is kernel.function& process.statement& timer.ms& %egin& end& 6tapset7
aliases. 2or more info look at the man page of Jstapprobes7.
5an"ler can ha!e:
filtering/conditionals 6if ... net7
control structures 6foreach& while7
$n the script you don.t need to declare the type of !aria%le. $t is inferred from the
contet. To make our life easier helper functions like pid& eecname& log etc are defined.
Look at the language reference guide for more info. $f you ha!e installed the package
then you can find it at J/usr/share/doc/systemtap,<version>/langref.pdfK.
$o% to 8un
stap ,e .LscriptM. N,c Ltarget programMO
stap script.stp N,c Ltarget programMO
stap ,l .Le!entPM.
Tapset 9ibraries
$n the eample shown earlier after pro%ing on read system call we printed the name of
the system call and the arguments passed to !ia JnameK and JargstrK.
This was possi%le %ecause in the one of the tapset li%rary
J/usr/share/systemtap/tapset/syscalls2.stpK& following is defined.
pro%e syscall.read 3 kernel.function65SySIread57.call Q&
kernel.function65sysIread57.call
9
name 3 5read5
fd 3 8fd
%ufIuaddr 3 8%uf
count 3 8count
argstr 3 sprintf65:d& :p& :d5& 8fd& 8%uf& 8count7
=
Tapsets pro!ide a%straction to common pro%e points and define functions& which you can
use in your script. They are not runna%le 6pro%e aliases not pro%es7.

:*amples
8 cat syscount.stp
glo%al syscalls
pro%e syscall.P 9 syscallsNnameO R3 > =
pro%e timer.s6>07 9
foreach6n in syscalls, limit @7
printf65:s 3 :d;n5& n& syscallsNnO7
delete syscalls
=
Sere we ha!e taken an associati!e array syscalls. ;ssociati#e array is a collection of
uni(ue keys< each key in the array has a !alue associated with it. Sere system call name
would %e an uni(ue key.
Whene!er a system call is called& we increment the !alue of corresponding !alue in the
array.Then after >0 sec we print the top @ system calls& which were called.
8 stap syscount.stp
read 3 >>?
poll 3 @@
ppoll 3 CB
setitimer 3 2C
write! 3 22
Let.s look at another script from which we want get the process name and pid of the
process who calls the maimum system calls. We also don.t want to take SystemTap
process which launches the script6stapio7& into consideration.
8 cat syscountIperIprocess.stp
glo%al syscalls
pro%e syscall.P 9
if 6eecname67 33 5stapio57
net
syscallsNeecname67& pid67O R3 >
=
pro%e timer.s6>07 9
foreach6Nprocname& idO in syscalls, limit @7
printf65:sN:dO 3 :d;n5& procname& id& syscallsNprocname& idO7
delete syscalls
=
To immediately return from the enclosing pro%e handled we use next statement.
8 stap syscountIperIprocess.stp
hald,addon,storN>0ECO 3 A0
sendmailN>>@EO 3 >C
rtkit,daemonN>A8EO 3 8
gdm,simple,greeN>AECO 3 8
gnome,power,manN>AE0O 3 E
We can do other interesting stuff like aggregation& getting a call graph& modifying a
kernel !aria%le in the running kernel etc. This all we co!er in net month.
8eferences
http://sourceware.org/systemtap/
http://sourceware.org/systemtap/wiki/LW2008SystemTapTutorial
http://sourceware.org/systemtap/wiki/Some-age"
action3)ttach2ileTdo3!iewTtarget3fosdem,stap.pdf
http://www.red%ooks.i%m.com/redpapers/pdfs/redpCC?B.pdf