SYSTEMTAP FOR SYSTEM ADMINISTRATORS

Heath Petty Sr. Technical Account Manager, Red Hat May 4th 2011

Systemtap sounds awesome, and looks like it can do a lot of cool things,
But how do I (non-programmer) use it? What cases would be appropriate? Why would it be better than other monitoring tools?

How do I use Systemtap?

Read the Systemtap Beginners Guide ...

Read the Systemtap Beginners Guide for details As a quick start, Install the following packages (use yum to grab all the necessary dependancies): systemtap systemtap-runtime kernel-devel-$(uname -r) kernel-debuginfo-$(uname -r) kernel-debuginfo-common-[arch]-$(uname -r)

But I don't want to install kernel-debuginfo on all my production systems (we'll get to that later)

Hello world is nice for testing, but lets do a more interesting test: stap -v -e 'probe vfs.read {printf("something read something from somewhere\n"); exit()}'

Now that we know it works, lets fill in some details, but how? Documentation time ! Man pages Red Hat Systemtap documentation Using stap -L Systemtap Wiki: http://sourceware.org/systemtap/wiki

# stap -L 'vfs.read' vfs.read file:long pos:long buf:long bytes_to_read:long dev:long devname:string ino:long name:string argstr:string $file:struct file* $buf:char* $count:size_t $pos:loff_t* # stap -L 'kernel.trace(*)' kernel.trace("block_bio_backmerge") $q:struct request_queue* $bio:struct bio* kernel.trace("block_bio_bounce") $q:struct request_queue* $bio:struct bio* kernel.trace("block_bio_complete") $q:struct request_queue* $bio:struct bio* ...

Now that we know systemtap is working, how can we get around installing kernel-debuginfo, compilers and other packages on your production machines? To accomplish this we can compile the systemtap script on a dev machine and then load that module on a production machine

Requirements: Dev machine must be same arch as the target system. The kernel-devel and kernel-debuginfo package versions must match the target system. Systemtap-runtime must be installed on the target system.

Example: stap -e 'probe vfs.read {exit()}' -m simple -p4 This will create the simple.ko file in the current directory. Copy this file to the target system and run: staprun simple.ko

Awesome, but how do I gather information from my script? I can't just sit there and watch it. Either direct output from a system script to a file, or use the handy flight recording functionality (the -F stap flag). Writing to a rotating log file: stap -F -o /tmp/name.log -S 1,2 <script> stap -F <script> I'll show how to use these options with our examples.

Where to next? Should I write my own systemtap scripts? How do I know what to do? This is the point where I give up on systemtap, because it seems really complicated Like learning a whole new programming language. There is a solution! There are numerous example scripts provided with systemtap that will address a large number of problems. There are even more online. Lets review a few of the examples and have some fun.

Sample 1. I really don't like spaces in filenames I'm old skool. /usr/share/doc/systemtap[ver]/examples/general/badnames.stp

Sample 2: iostat lies sometimes Lesson: the io examples might be better for troubleshooting I/O issues. /usr/share/doc/systemtap[ver]/examples/io/disktop.stp

Sample 3: Lets find who is leaking memory /usr/share/doc/systemtap[ver]/examples/memory/mmanonpage.stp

Sample 4: See if network buffers get full.

/usr/share/doc/systemtap1.2/examples/network/sk_stream_wait_memory.stp

I have more examples if time permits, if not, then we can take a look at them in Support Central