Exploring Splunk

Exploring Splunk
SEARCH PROCESSING LANGUAGE (SPL)

PRIMER AND COOKBOOK
By David Carasso, Splunks Chief Mind
CITO
Research
New York, NY
Exploring Splunk, by David Carasso
Copyright
2012 by Splunk Inc.

All rights reserved. Printed in the United States of America.
Authorization to photocopy items for internal or personal use is granted
by Splunk, Inc. No other copying may occur without the express written
consent of Splunk, Inc.
Published by CITO Research, 1375 Broadway, Fl3, New York, NY 10018.
Editor/Analyst: Dan Woods, Deb Cameron
Copyeditor: Deb Cameron
Production Editor: Deb Gabriel
Cover: Splunk, Inc.
Graphics: Deb Gabriel
First Edition: April 2012
While every precaution has been taken in the preparation of this book,
the publisher and author assume no responsibility for errors or omissions
or for damages resulting from the use of the information contained herein.
ISBN: 978-0-9825506-7-0; 0-9825506-7-7
Disclaimer
This book is intended as a text and reference book for reading purposes
only. The actual use of Splunks software products must be in accordance
with their corresponding software license agreements and not with any-
thing written in this book. The documentation provided for Splunks soft-
vure products, und not ths book, s the dentve source or normuton
on how to use these products.
Although great care has been taken to ensure the accuracy and timeliness
of the information in this book, Splunk does not give any warranty or
guarantee of the accuracy or timeliness of the information and Splunk does
not assume any liability in connection with any use or result from the use
of the information in this book. The reader should check at docs.splunk.
com or dentve descrptons o Spunk's eutures und unctonuty.
Table of Contents
Preface
About This Book i
Whats In This Book? ii
Conventions ii
Acknowledgments iii
PART I: EXPLORING SPLUNK
1 The Story of Splunk
Splunk to the Rescue in the Datacenter 3
Splunk to the Rescue in the Marketing Department 4
Approaching Splunk 5
Splunk: The Company and the Concept 7
How Splunk Mastered Machine Data in the Datacenter 8
Operational Intelligence 9
Operational Intelligence at Work 11
2 Getting Data In
Machine Data Basics 13
Types of Data Splunk Can Read 15
Splunk Data Sources 15
Downloading, Installing, and Starting Splunk 15
Bringing Data in for Indexing 17
Understanding How Splunk Indexes Data 18
3 Searching with Splunk
The Search Dashboard 23
SPL: Search Processing Language 27
Pipes 27
Implied AND 28
top user 28
elds percent 28
The search Command 29
Tips for Using the search Command 30
Subsearches 30
4 SPL: Search Processing Language
Sorting Results 33
sort 33
Filtering Results 35
where 35
dedup 36
head 38
Grouping Results 39
transaction 39
Reporting Results 41
top 41
stats 43
chart 45
timechart 47
Filtering, Modifying, and Adding Fields 48
elds 49
replace 50
eval 51
rex 52
lookup 53
5 Enriching Your Data
Using Splunk to Understand Data 55
Identifying Fields: Looking at the Pieces of the Puzzle 56
Exploring the Data to Understand its Scope 58
Preparing for Reporting and Aggregation 60
Visualizing Data 65
Creating Visualizations 65
Creating Dashboards 67
Creating Alerts 68
Creating Alerts through a Wizard 68
Tuning Alerts Using Manager 71
Customizing Actions for Alerting 74
The Alerts Manager 74
PART II: RECIPES
6 Recipes for Monitoring and Alerting
Monitoring Recipes 79
Monitoring Concurrent Users 79
Monitoring Inactive Hosts 80
Reporting on Categorized Data 81
Comparing Todays Top Values to Last Months 82
Finding Metrics That Fell by 10% in an Hour 84
Charting Week Over Week Results 85
Identify Spikes in Your Data 86
Compacting Time-Based Charting 88
Reporting on Fields Inside XML or JSON 88
Extracting Fields from an Event 89
Alerting Recipes 90
Alerting by Email when a Server Hits a Predened Load 90
Alerting When Web Server Performance Slows 91
Shutting Down Unneeded EC2 Instances 91
Converting Monitoring to Alerting 92
7 Grouping Events
Introduction 95
Recipes 97
Unifying Field Names 97
Finding Incomplete Transactions 97
Calculating Times within Transactions 99
Finding the Latest Events 100
Finding Repeated Events 101
Time Between Transactions 102
Finding Specic Transactions 104
Finding Events Near Other Events 107
Finding Events After Events 108
Grouping Groups 109
8 Lookup Tables
Introduction 113
lookup 113
inputlookup 113
outputlookup 113
Further Reading 114
Recipes 114
Setting Default Lookup Values 114
Using Reverse Lookups 114
Using a Two-Tiered Lookup 116
Using Multistep Lookups 116
Creating a Lookup Table from Search Results 117
Appending Results to Lookup Tables 117
Using Massive Lookup Tables 118
Comparing Results to Lookup Values 120
Controlling Lookup Matches 122
Matching IPs 122
Matching with Wildcards 123
Appendix A: Machine Data Basics
Application Logs 126
Web Access Logs 126
Web Proxy Logs 127
Call Detail Records 127
Clickstream Data 127
Message Queuing 128
Packet Data 128
Conguration Files 128
Database Audit Logs and Tables 128
File System Audit Logs 128
Management and Logging APIs 129
OS Metrics, Status, and Diagnostic Commands 129
Other Machine Data Sources 129
Appendix B: Case Sensitivity
Appendix C: Top Commands
Appendix D: Top Resources
Appendix E: Splunk Quick Reference Guide
CONCEPTS 137
Overview 137
Events 137
Sources and Sourcetypes 138
Hosts 138
Indexes 138
Fields 138
Tags 138
Event Types 139
Reports and Dashboards 139
Apps 139
Permissions/Users/Roles 139
Transactions 139
Forwarder/Indexer 140
SPL 140
Subsearches 141
Relative Time Modiers 141
COMMON SEARCH COMMANDS 142
Optimizing Searches 142
SEARCH EXAMPLES 143
EVAL FUNCTIONS 146
COMMON STATS FUNCTIONS 151
REGULAR EXPRESSIONS 152
COMMON SPLUNK STRPTIME FUNCTIONS 153
i
Preface
Splunk Enterprise Software (Splunk) is probably the single most power-
ful tool for searching and exploring data that you will ever encounter. We
wrote this book to provide an introduction to Splunk and all it can do.
This book also serves as a jumping off point for how to get creative with
Splunk.
Splunk is often used by system administrators, network administrators,
and security gurus, but its use is not restricted to these audiences. There is
a great deal of business value hidden away in corporate data that Splunk
can liberate. This book is designed to reach beyond the typical techie
reader of OReilly books to marketing quants as well as everyone inter-
ested in the topics of Big Data and Operational Intelligence.
About This Book
The central goal of this book is to help you rapidly understand what
Splunk is and how it can help you. It accomplishes this by teaching you
the most important parts of Splunks Search Processing Language (SPL).
Splunk can help technologists and businesspeople in many ways. Dont
expect to learn Splunk all at once. Splunk is more like a Swiss army knife,
a simple tool that can do many powerful things.
Now the question becomes: How can this book help? The short answer is
by quickly giving you a sense of what you can do with Splunk and point-
ers on where to learn more.
But isnt there already a lot of Splunk documentation? Yes:
- If you check out http://docs.splunk.com, you v nd muny munuus
with detailed explanations of the machinery of Splunk.
- If you check out http://splunkbase.com, you v nd u seurchube
database of questions and answers. This sort of content is invaluable
when you know a bit about Splunk and are trying to solve common
problems.
This book falls in between these two levels of documentation. It offers
a basic understanding of Splunks most important parts and combines it
with solutions to real-world problems.
Whats In This Book?
Chapter 1 tells you what Splunk is and how it can help you.
Chapter 2 discusses how to download Splunk and get started.
Chapter 3 discusses the search user interface and searching with Splunk.
Chapter 4 covers the most commonly used parts of the SPL.
Chapter 5 explains how to visualize and enrich your data with knowl-
edge.
Chapter 6 covers the most common monitoring and alerting solutions.
Chapter 7 covers solutions to problems that can be solved by grouping
events.
Chapter 8 covers many of the ways you can use lookup tables to solve
common problems.
If you think of Part I (chapters 1 through 5) as a crash course in Splunk,
Part II (chapters 6 through 8) shows you how to do some advanced ma-
neuvers by putting it all together, using Splunk to solve some common
and interesting problems. By reviewing these recipesand trying a few
youll get ideas about how you can use Splunk to help you answer all the
mysteries of the universe (or at least of the data center).
The appendices round out the book with some helpful information. Ap-
pendix A provides an overview of the basics of machine data to open
your eyes to the possibilities and variety of Big Data. Appendix B provides
a table on what is and isnt case-sensitive in Splunk searches. Appendix C
provides a glimpse into the most common searches run with Splunk (we
gured ths out usng Spunk, by the vuy). Appendx D oers ponters to
some of the best resources for learning more about Splunk. Appendix E is
a specially designed version of the Splunk Reference card, which is the
most popular educational document we have.
Conventions
As you read through this book, youll notice we use various fonts to call
out certain elements:
- UI elements appear in bold.
- Communds und ed numes ure n !"#$%&#% ()*%+.
If you are told to select the Y option from the X menu, thats written con-
cisely as select X Y.
iii
Acknowledgments
This book would not have been possible without the help of numerous
people at Splunk who gave of their time and talent. For carefully review-
ing drafts of the manuscript and making invaluable improvements, wed
like to thank Ledion Bitincka, Gene Hartsell, Gerald Kanapathy, Vishal
Patel, Alex Raitz, Stephen Sorkin, Sophy Ting, and Steve Zhang, PhD; for
generously giving interview time: Maverick Garner; for additional help:
Jessica Law, Tera Mendonca, Rachel Perkins, and Michael Wilde.
PART I
EXPLORING SPLUNK
3
1 The Story of Splunk
Splunk is a powerful platform for analyzing machine data, data that ma-
chines emit in great volumes but which is seldom used effectively. Ma-
chine data is already important in the world of technology and is becom-
ing increasingly important in the world of business. (To learn more about
machine data, see Appendix A.)
The fastest way to understand the power and versatility of Splunk is to consider
two scenarios: one in the datacenter and one in the marketing department.
Splunk to the Rescue in the Datacenter
Its 2 AM on Wednesday. The phone rings. Your boss is calling; the web-
site is down. Why did it fail? Was it the web servers, the applications, the
database servers, a full disk, or load balancers on the fritz? Hes yelling at
you |o |x || nov. l|s ta|n|ng. \oute |teal|ng ou|.
Re|ax. \ou dep|oyed Sp|unl yes|etday.
\ou s|at| up Sp|unl. Itom one p|ace, you can seatch |he |og ||es |tom
a|| yout veb setvets, da|abases, |teva||s, tou|ets, and |oad ba|ancets,
as ve|| as seatch con|guta||on ||es and da|a |tom a|| yout o|het dev|ces,
opeta||ng sys|ems, ot app||ca||ons o| |n|etes|. (1h|s |s |tue no ma||et hov
many da|acen|ets ot c|oud ptov|dets |hese may be sca||eted actoss.)
\ou |ool a| a gtaph o| veb setvet |ta||c |o see vhen |he ptob|em hap-
pened. | S:6! IM, ettots on |he veb setvets sp|led dtama||ca||y. \ou
|hen |ool a| |he |op 16 pages v||h ettots. 1he home page |s olay. 1he
seatch page |s olay. h, |he shopp|ng cat| |s |he ptob|em. S|at||ng a|
S:6!, evety teques| |o |ha| page ptoduced an ettot. 1h|s |s cos||ng mon-
eypteven||ng sa|es and dt|v|ng avay cus|ometsand || mus| be |xed.
\ou lnov |ha| yout shopp|ng cat| te||es on an ecommetce setvet con-
nected to a database. A look at the logs shows the database is up. Good.
Ie|s |ool a| |he ecommetce setvet |ogs. | S:6! IM, |he ecommetce setv-
er starts saying it cannot connect to the database server. You then search
|ot changes |o |he con|guta||on ||es and see |ha| someone changed a
network setting. You look closer; it was done incorrectly. You contact the
petson vho made |he change, vho to||s || bacl, and |he sys|em s|at|s
working again.
|| o| |h|s can |ale |ess |han S m|nu|es because Sp|unl ga|heted a|| o| |he
te|evan| |n|otma||on |n|o a cen|ta| |ndex |ha| you cou|d tap|d|y seatch.
Exploring Splunk
4
Splunk to the Rescue in the Marketing Department
\ou votl |n |he ptomo||ons depat|men| o| a |atge te|a||et. \ou |une |he
seatch eng|ne op||m|za||on and ptomo||ons |ot yout ptoduc|s |o op||m|ze
|he y|e|d o| |ncom|ng |ta||c. Ias| veel, |he guys |tom |he da|acen|et |n-
s|a||ed a nev Sp|unl dashboatd |ha| shovs (|ot |he pas| hout, day, and
veel) a|| |he seatch |etms used |o |nd yout s||e.
Iool|ng a| |he gtaph |ot |he |as| |ev houts, you see a sp|le 26 m|nu|es
ago. Seatches |ot yout company name and yout |a|es| ptoduc| ate vay
up. \ou checl a tepot| on |op te|ett|ng IRIs |n |he pas| hout and Sp|unl
shows that a celebrity tweeted about the product and linked to your
home page.
\ou |ool a| ano|het gtaph |ha| shovs pet|otmance o| |he mos| |tequen||y
visited pages. The search page is overloaded and slowing down. A huge
ctovd o| peop|e |s com|ng |o yout s||e bu| can| |nd |he ptoduc| |hey ate
looking for, so they are all using search.
\ou |og on |o yout s||es con|en| managemen| sys|em and pu| a ptomo-
||ona| ad |ot |he nev ptoduc| a| |he cen|et o| |he home page. \ou |hen
go bacl and |ool a| |he |op pages. Seatch |ta||c s|at|s |o dtop, and |ta||c
|o |he nev ptoduc| page s|at|s |o t|se, and so does |ta||c |o |he shopp|ng
cart page. You look at the top 10 products added to the cart and the top
10 products purchased; the new product tops the list. You send a note
|o |he IR depat|men| |o |o||ov up. lncom|ng |ta||c |s nov convet||ng |o
sa|es |ns|ead o| |tus|ta||on, exac||y vha| you van| |o happen. \out ab||||y
|o male |he mos| o| an un|oteseen oppot|un||y vas made poss|b|e by
Sp|unl. \out nex| s|ep |s |o male sute |ha| you have enough o| |ha| ptod-
uc| |n s|ocl, a gtea| ptob|em |o have.
1hese |vo examp|es shov hov Sp|unl can ptov|de a de|a||ed v|ndov
|n|o vha| |s happen|ng |n yout mach|ne da|a. Sp|unl can a|so tevea|
h|s|ot|ca| |tends, cotte|a|e mu|||p|e soutces o| |n|otma||on, and he|p |n
thousands of other ways.
Chapter 1: The Story of Splunk
5
Approaching Splunk
As you use Spunk to unsver questons, you' nd thut you cun breuk the
task into three phases.
- First, identify the data that can answer your question.
- Second, transform the data into the results that can answer your
question.
- Third, display the answer in a report, interactive chart, or graph to
make it intelligible to a wide range of audiences.
Begin with the questions you want to answer: Why did that system fail?
Why is it so slow lately? Where are people having trouble with our web-
site? As you master Splunk, it becomes more obvious what types of data
and searches help answer those questions. This book will accelerate your
progress to mastery.
The question then becomes: Can the data provide the answers? Often,
when we begin an analysis, we dont know what the data can tell us. But
Splunk is also a powerful tool for exploring data and getting to know it.
You can discover the most common values or the most unusual. You can
summarize the data with statistics or group events into transactions, such
as all the events that make up an online hotel reservation across systems
o record. You cun creute vorkovs thut begn vth the vhoe dutu set,
then ter out rreevunt events, unuyzng vhut's et. 1hen, perhups, udd
some information from an external source until, after a number of simple
steps, you have only the data needed to answer your questions. Figure
1-1 shows, in general, the basic Splunk analysis processes.
Exploring Splunk
6
I|gute 11. \otl|ng v||h Sp|unl
1
2
.
1
.
1
.
1
4
0

-

-

[
0
1
/
A
u
g
/
2
0
0
9
:
0
9
:
3
7
:
0
1

-
0
7
0
0
]

"
G
E
T

/
h
o
m
e
/
t
h
e
m
e
s
/
C
o
m
B
e
t
a
/
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
n
.
g
i
f

H
T
T
P
/
1
.
1
"

3
0
4

-

"
h
t
p
:
/
/
w
e
b
d
e
v
:
2
0
0
0
/
h
o
m
e
/
i
e
x
.
p
h
p
"

"
M
o
z
i
l
l
a
/
5
.
0
1
2
.
1
.
1
.
1
4
0

-

-

[
0
1
/
A
u
g
/
2
0
0
9
:
0
9
:
3
7
:
0
1

-
0
7
0
0
]

"
G
E
T

/
h
o
m
e
/
t
h
e
m
e
s
/
C
o
m
B
e
t
a
/
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
n
.
g
i
f

H
T
T
P
/
1
.
1
"

3
0
4

-

"
h
t
p
:
/
/
w
e
b
d
e
v
:
2
0
0
0
/
h
o
m
e
/
i
e
x
.
p
h
p
"

"
M
o
z
i
l
l
a
/
5
.
0
1
2
.
1
.
1
.
1
4
0

-

-

[
0
1
/
A
u
g
/
2
0
0
9
:
0
9
:
3
7
:
0
1

-
0
7
0
0
]

"
G
E
T

/
h
o
m
e
/
t
h
e
m
e
s
/
C
o
m
B
e
t
a
/
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
n
.
g
i
f

H
T
T
P
/
1
.
1
"

3
0
4

-

"
h
t
p
:
/
/
w
e
b
d
e
v
:
2
0
0
0
/
h
o
m
e
/
i
e
x
.
p
h
p
"

"
M
o
z
i
l
l
a
/
5
.
0
PHASE II
1
2
.
1
.
1
.
1
4
0

-

-

[
0
1
/
A
u
ggg
/
2
0
0
9
:
0
9
:
3
7
:
0
1

-
0
7
0
0
]

-
0
"
G
E
T

/
h
o
m
e
/
t
h
e
m
e
s
/
C
o
m
BBB
"
G
E
T

/
h
o
m
e
/
t
h
e
m
e
s
/
C
n
.
ggg
e
t
a
/
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
n
.
g
"
h
ttt
i
f

H
T
T
P
/
1
.
1
"

3
0
4

-

"
h
0
0
0
/
h
o
m
e
/
iii
0
0
0
/
h
o
m
v
:
2
v
:
2
w
e
b
d
e
w
e
b
d
e
v
:
/
/
w
/
/
pp
:
e
x
.
p
h
p
"

"
M
o
z
i
l
l
a
/
5
.
0
l
a
"
.
111
111
..
111
44
0

-

-

[
0
1
///
22
0
00
99
::
00
999
::
333
7
:
0
1

-
m
t
h
1
ee
vvvv
2
0
/
h
o
m
e
vvvv
:
22
0
pp
:::
//
/
www
ee
p
:::
//
//
www
i
ll
ll
a
/
5
l
ee
hh
ppp
"
"
M
1
2
.
1
.
1
.
1
4
0

-

-

[
0
1
/
A
u
ggg
0
1

-
0
7
0
0
]

-
0
/
2
0
0
9
:
0
9
:
3
7
:
0
1
/
C
o
m
BBB
t
h
e
m
e
s
/
C
o
"
/
/
"
G
E
T

/
h
o
m
e
/
t
n
.
ggg
e
t
a
/
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
n
.
g
"
h
ttt
i
f

H
T
T
P
/
1
.
1
"

3
0
4

-

"
h
0
0
0
h
o
m
e
/
iii
0
0
0
/
h
o
v
:
2
v
:
2
w
e
b
d
e
w
e
b
d
e
v
:
/
/
w
/
/
pp
:
e
x
.
p
h
p
"

"
M
o
z
i
l
l
a
/
5
.
0
l
a
"
.
111
111
..
111
44
00

-

-

[
0
1
///
22
0
00
99
::
00
999
::
333
7
:
0
1

-
t
h
m
1
ee
vvvv
2
0
/
h
o
m
e
vvvv
:
22
0
pp
:::
//
/
www
ee
p
:::
//
//
www
i
ll
ll
a
/
5
l
ee
hh
ppp
"
"
M
1
2
.
1
.
1
.
1
4
0

-

-

[
0
1
/
A
u
ggg
0
1

-
0
7
0
0
]

-
0
/
2
0
0
9
:
0
9
:
3
7
:
0
1
h
e
m
e
s
/
C
o
m
BBB
t
h
e
m
e
s
/
C
o
/
/
"
G
E
T

/
h
o
m
e
/
t
n
.
ggg
e
t
a
/
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
n
.
g
"
h
ttt
i
f

H
T
T
P
/
1
.
1
"

3
0
4

-

"
h
p
:
/
/
w
e
b
d
e
v
:
2
0
0
0
/
h
o
m
e
/
iii
p
:
/
/
w
e
b
d
e
v
:
2
0
0
0
/
h
o
m
e
x
.
p
h
p
"

"
M
o
z
i
l
l
a
/
5
.
0
l
a
.
111
111
..
111
44
00

-

-

[
0
1
///
22
0
00
99
::
00
999
::
333
7
:
0
1

-
t
h
m
1
pp
:::
//
/
www
ee
ee
vvvv
2
0
/
h
o
m
p
:::
//
//
www
e
vvvv
:
22
0
ee
hh
ppp
"
"
M
i
ll
ll
a
/
5
l
PHASE I Gather data from as many sources as necessary
PHASE III
1
00

-
0
/
2
/
222
0
0
9
:
0
9
:
3
7
:
0
1

-
0
7
0
0
]
"
GGG
E
T

/
h
o
m
e
/
t
h
e
m
e
s
/
C
o
m
B
G
e
t
e
tttt
a
/
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
n
.
g
t
i
fff

H
T
T
P
/
1
.
1
"

3
0
4

-

"
h
t
f
2
/
2
"
G
E
T
e
t
a
/
i
m
i
f

H
T
T
d
ee
p
:
/
p
:
/
/
w
e
/
w
e
bbb
d
e
z
iii
e
x
.
p
h
p
h
"""

"
M
o
z
i
aaaa
[
0
1
/
A
u
g
a
1
2
.
1
.
1
..
1
4
0

-

-

p
p
-
0
7
0
0
/
2
0
/
2
0
0
9
:
0
9
:
000
9
:
0
9
:
3
7
:
3
7
:
0
1
0
1
h
e
"
G
.
g
t
n
t
n
e
t
-

"
h
t
33
i
f
0
/
h
/
//
i
p
:
/
/
w
e
bb
d
e
v
:
2
0
0
0
/
h
o
m
e
/
i
e
x
.
p
h
p
""

"
M
o
z
i
l
l
a
/
5
.
0
-
[
0
1
/
A
u
g
1
2
.
1
.
1
..
1
4
0

-

-

/
2
0
0
9
:
00
9
:
"
G
E
T

/
e
t
a
/
i
m
T
TT
i
f

H
T
T
//
/
w
e
/
w
b
d
e
v
:
2
0
0
0
/
h
o
m
e
/
i
p
:
/
/
w
e
x
.
e
p
h
p
"

"
M
o
z
i
l
l
a
/
5
.
0
e
x
-

[
0
1
/
A
u
g
1
2
/
2
0
0
9
:
00
9
:
3
7
:
0
1

-
0
7
0
0
]

0
7
0
0
]
"
G
E
T

/
hh
o
m
e
/
t
h
e
mm
e
s
/
C
o
m
B
t
n
e
t
333
i
f
0
p
: 22
1
1
1
1
1
4
0
t
a
/
i
m
aa
g
eeee
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
/
ttt
f

H
T
T
PPPPPP
/
1
PPPP
/
P
/
1
P
/
1
P
/
P
/
1
P
/
1
P
/
P
/
1
/
1
///
1
//
1
/
1
/
1
/
1
/
1
/
1
/
11
/
111111111111
.
1
.
1
.
1
.
1
.
1
.
1111111111
"
1
"
1
.
111111
""""""""""""""
/
/
w
e
/
w
e
/
w
e
/
w
e
/
w
e
w
e
w
eee
bbbbbbbbbbb
d
e
b
d
e
b
d
e
b
d
b
d
b
d
b
d
b
d
b
d
b
d
v
:
2
0
2
0
0
9
:
00
9
:
3
7
:
0
G
EE
T

/
T

/
T

/
T

/
T

//
hhh
o
m
e
/
t
h
t
a
//
i
m
/
i
m
i
m
i
m
i
m
i
mmmmm
aaaaaaaaaaa
g
e
a
g
e
a
g
e
a
g
e
a
g
e
a
g
e
a
g
e
a
g
e
a
g
e
aa
g
e
a
g
e
g
eeee
s
/
s
/
s
/
s
/
s
/
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
s
/
b
/
tt
f

H
T
T
PP
/
1
P
/
1
/
1
/
1
/
1
/
1111111
.
1
"
.
1
"
.
1
"
.
1
"
.
1
"
.
1
"
.
1
"
.
11
""""""
/
/
w
e
bb
d
e
v
:
2222
00
x
.
p
h
p
""

"
M
o
z
M
o
z
M
o
z
M
o
z
M
o
M
o
M
o
i
l
2
..
1
.
1
1
.
1
1
.
1
1
.
1
1
.
11111
......
1
.
1
4
.
1
4
.
1
4
.
1
4
.
1
4
.
1
4
.
1
4
1
4
1
4
1
44
00

0

0

-
0

-
0

-
0

-
0

-
0

-
0

-
0

-
0

-
0

--

-
2
0
0
9
:
00
9
:
3
7
:
3
7
:
3
7
3
77
0
1
Visualize or review the data to gain insight
Transform the data
into answers
----------
-

[
0
1
/
A
u
g
0000000000
9
:
0
9
:
3
7
:
0
1

-
0
7
0
0
]
TTTTTTTTTT

/
h
o
m
e
/
t
h
e
m
e
s
/
C
ooo
mmmmmmm
//////////
i
m
a
g
e
s
/
b
t
n
_
l
o
g
i
nnn
.
HHHHHHHHHH
T
T
P
/
1
.
1
"

3
0
4

-
"
hhhhh
sourcetype
syslog
syslog
other-source
syslog
syslog
syslog
other-source
syslog
other-source
<events>
raw
ERROR
ERROR
WARNING
WARNING
ERROR
IP address
12.1.1.002
12.1.1.140
12.1.1.140
12.1.1.002
12.1.1.43
<felds...>
Social Media
Data
Credit Card
Data
7
Splunk: The Company and the Concept
The real excitement most people feel about Splunk comes from its ability
to help solve the complex and recurring problems that Splunk customers
have always had. The story of Splunk began in 2002, when cofounders
Erik Swan and Rob Das started looking around for their next challenge.
Erik and Rob had done a couple of startups together and were looking
for an idea for a new venture, so they started talking to companies about
their problems.
Erik and Rob asked prospective customers, How do you solve prob-
lems in your infrastructure? Over and over again, Erik and Rob heard
about practitioners experiences trying to troubleshoot IT problems and
retrieve data by traditional means. The data was too spread out; it was
hard to bring it all together and make sense of it. Everyone was attempting
to sove probems by munuuy porng over og es, sometmes vrtng
scripts to help them along. The homegrown scripts were brittle, the peo-
ple who wrote them sometimes left the company and took their expertise
with them, and every new attempt to explore an issue would result in
nger-pontng, buck-pussng, und scrpt rebudng, vth heuvy-duty cus-
tom assistance from the IT department. These practitioners told Splunks
founders that solving infrastructure problems was like slowly crawling
around in caves (their datacenters) with pickaxes, poor lighting, and lim-
ited navigational power (old scripts and log management technologies).
In short, it was like spelunkingand so, the name Splunk was born.
Cven the dcuty o dgtu speunkng, the ony uternutve uvuube to
these people was to search the Web to see if other companies had similar
problems and had posted solutions online. The founders were stunned
that people were spending money on this widely acknowledged issue,
and yet no one had stepped up to solve the problem. Erik and Rob asked
themselves, Why couldnt searching IT data be as easy and intuitive as
a Google search?
1he rst vson o Spunk vus umed ut mukng t much euser to ussembe
and analyze the data needed to run and troubleshoot a datacenter or
large computing or networking environment. The mission of Splunk was
to combine the ease of a web search with the power of the laborious,
homegrown methods IT professionals were using to troubleshoot problems.
Exploring Splunk
8
Lrk und Rob rused undng und the rst verson o Spunk debuted ut
LinuxWorld
2005. The product was a huge hit and immediately went vi-
ral, spurred on by its availability as a free download. Once downloaded,
Splunk began solving broad range of unimagined customer problems and
spread from department to department and from company to company.
When users asked management to purchase it, they could already point
to a record of solving problems and saving time with Splunk.
Originally conceived to help IT and datacenter managers troubleshoot
technical problems, Splunk has grown to become an extremely useful
platform for all kinds of business users because it enables them to search,
collect, and organize data in a far more comprehensive, far less labor-in-
tensive way than traditional databases. The result is new business insights
and operational intelligence that organizations have never had before.
How Splunk Mastered Machine Data in the Datacenter
1he rst puce thut Spunk took hod, nuturuy, vus the dutucenter, vhch
is awash in machine data. Splunk became popular with system admin-
istrators, network engineers, and application developers as an engine to
quickly understand (and increase the usefulness of) machine data. But
why did they like it so much? An example helps not only explain Splunks
early popularity but also helps us understand the nature of machine data,
which is central to the larger value that Splunk brings to the business
world.
In most computing environments, many different systems depend on
each other. Monitoring systems send alerts after something goes wrong.
For example, the key web pages of a site may depend on web servers, ap-
pcuton servers, dutubuse servers, e systems, oud buuncers, routers,
application accelerators, caching systems, and so on. When something
goes wrong in one of these systems, say a database, alarms may start
sounding at all levels, seemingly at once. When this happens, a system
udmnstrutor or uppcuton specust must nd the root cuuse und x
t. 1he probem s thut the og es ure spreud ucross mutpe muchnes,
sometimes in different time zones, and contain millions of entries, most
of which have nothing to do with the problem. In addition, the relevant
recordsthe ones that indicate some failure of the systemtend to ap-
peur u ut once. 1he chuenge then s to nd the probem thut sturted t
all. Lets look at how Splunk helps do this.
- Splunk begins with indexing, which means gathering all the data
from diverse locations and combining it into centralized indexes.
Before Splunk, system administrators would have had to log in to
9
many different machines to gain access to all the data using far less
powerful tools.
- Using the indexes, Splunk can quickly search the logs from all
servers and hone in on when the problem occurred. With its speed,
scale, and usability, Splunk makes determining when a problem oc-
curred that much faster.
- Splunk can then drill down into the time period when the problem
rst occurred to determne ts root cuuse. Aerts cun then be creuted
to head the issue off in the future.
y ndexng und uggregutng og es rom muny sources to muke them
centrally searchable, Splunk has become popular among system admin-
istrators and other people who run technical operations for businesses
around the world. Security analysts use Splunk to sniff out security vul-
nerubtes und uttucks. System unuysts use Spunk to dscover ne-
ciencies and bottlenecks in complex applications. Network analysts use
Spunk to nd the cuuse o netvork outuges und bundvdth bottenecks.
This discussion brings up several key points about Splunk:
- Crcating a ccntraI rcpository is vitaI: One of the major victories
of Splunk is the way that diverse types of data from many different
sources are centralized for searching.
- SpIunk convcrts data into answcrs: Spunk heps you nd the n-
sights that are buried in the data.
- SpIunk hcIps you undcrstand thc structurc and mcaning of data:
The more you understand your data, the more youll see in it.
Splunk also helps you capture what you learn to make future investi-
gations easier and to share what youve learned with others.
- VisuaIization cIoscs thc Ioop: All that indexing and searching pays
off when you see a chart or a report that makes an answer crystal
clear. Being able to visualize data in different ways accelerates un-
derstanding and helps you share that understanding with others.
Operational Intelligence
Because almost everything we do is assisted in some way by technology,
the information collected about each of us has grown dramatically. Many
of the events recorded by servers actually represent behavior of custom-
ers or purtners. Spunk customers gured out eury on thut veb server
access logs could be used not only to diagnose systems but also to better
understand the behavior of the people browsing a website.
Exploring Splunk
10
Splunk has been at the forefront of raising awareness about operational
intelligence, a new category of methods and technology for using ma-
chine data to gain visibility into the business and discover insights for IT
and the entire enterprise. Operational intelligence is not an outgrowth of
business intelligence (BI), but a new approach based on sources of infor-
mation not typically within the purview of BI solutions. Operational data
is not only incredibly valuable for improving IT operations, but also for
yielding insights into other parts of the business.
Operational intelligence enables organizations to:
- Usc machinc data to gain a dccpcr undcrstanding of thcir custom-
crs: For example, if you just track transactions on a website, you
see what people bought. But by looking closely at the web server
logs you can see all the pages they looked at before they purchased,
and, perhaps even more important for the bottom line, you can see
the pages that the people who didnt buy looked at. (Remember our
new product search example from the intro?)
- RcvcaI important pattcrns and anaIytics dcrivcd from corrcIat-
ing cvcnts from many sourccs: When you can track indicators of
consumer behavior from websites, call detail records, social media,
and in-store retail transactions, a far more complete picture of the
customer emerges. As more and more customer interactions show
up in machine data, more can be learned.
- Rcducc thc timc bctwccn an important cvcnt and its dctcction:
Machine data can be monitored and correlated in real time.
- lcvcragc Iivc fccds and historicaI data to makc scnsc of what is
happcning now, to nd trcnds and anomaIics, and to makc morc
informcd dccisions bascd on that information: For example, the
truc creuted by u veb promoton cun be meusured n reu tme
and compared with previous promotions.
- DcpIoy a soIution quickIy and dcIivcr thc cxibiIity nccdcd by
organizations today and in thc futurc-that is, thc abiIity to pro-
vidc ad hoc rcports, answcr qucstions, and add ncw data sourccs:
Splunk data can be presented in traditional dashboards that allow
users to explore the events and keep asking new questions.
11
Operational Intelligence at Work
Spunk does somethng thut no other product cun: ecenty cupture und
analyze massive amounts of unstructured, time-series textual machine
data. Although IT departments generally start out using Splunk to solve
technically esoteric problems, they quickly gain insights valuable else-
where in their business.
Using machine data in Splunk helps solve vexing business problems.
Here are a few examples:
- An operations team implemented a cloud-delivered customer-facing
application and used Splunk for diagnostics. They soon realized they
could track user statistics and better plan capacitya metric with
profound business implications.
- \eb server truc ogs cun be used to truck shoppng curts beng
ed und ubundoned n reu tme. 1he murketng depurtment cun
use this information to determine where consumers are getting stuck
and what types of purchases are being abandoned so that any prob-
ems cun be xed rght uvuy und promotons cun turget tems thut
are abandoned.
- Organizations using Splunk to monitor applications for trouble-
shooting have realized that they can easily provide views to their
rst-ne support teum to hunde customer cus drecty, versus escu-
lating those calls to expensive engineering resources.
- A major utility company was able to eliminate costly software main-
tenance fees by replacing six other monitoring and diagnostic tools
with Splunk, while enhancing their NERC and SOX compliance
efforts.
- A major public media organization reduced the time it took to
capture critical web analytics from months to hours. They were also
able to track their digital assets with a granularity and accuracy that
they couldnt have otherwise, resulting in better royalty accounting
and content marketing.
- A taco fast-food restaurant connected its points of sale (POS) to
Splunk, and within an hour, business analysts were able to begin
answering questions like, How many people are buying tacos in
the midnight-to-2 AM period, in this geography, during this time of
the year?
Exploring Splunk
12
Ultimately, operational intelligence enables organizations to ask the right
questions, leading to answers that deliver business insights, using com-
binations of real-time and historical data, displayed in easily digestible
dashboards and graphical tools.
Theres a reason for the trend toward calling machine data big data. Its
big, its messy, and in there, buried somewhere, is the key to the future of
your business. Now lets move on to Chapter 2, where youll learn how to
get dutu nto Spunk und sturt ndng the god hdden n your dutu.
13
2 Getting Data In
Chapter 1 provided an introduction to Splunk and described how it can
help you. Now lets take the next step in your journey: getting your data
into Splunk.
This chapter covers installing Splunk, importing your data, and a bit about
how the data is organized to facilitate searching.
Machine Data Basics
Splunks mission is to make machine data useful for people. To give you
some context, its worth reviewing a few basics about machine data and
how Splunk keeps track of it.
People who create systems (such as web servers or load balancers or
video games or social media platforms) also specify the information those
systems vrte to og es vhen they ure runnng. 1hs normuton (the
muchne dutu n the og es) s vhut peope usng the systems cun use to
understand what those systems are doing as they run (or fail to run). For
exumpe, the og e output or u hypothetcu cock uppcuton mght
look like this:
,!%)"#- %)!./* $-012 3-402 +-542 *-672 3"-472 8-6455
,!%)"#- %)!./* $-092 3-402 +-542 *-672 3"-472 8-6455
,!%)"#- %)!./* $-0:2 3-402 +-542 *-672 3"-472 8-6455
,!%)"#- %)!./* $-442 3-4;2 +-542 *-672 3"-472 8-6455
Every time the clock ticks, it logs the action and the time that the action
occurred. If you were really going to keep track of the clock, in addition
to the fact that it ticked, the log might also include other useful infor-
mation: the battery level, when an alarm was set, turned on or off, or
soundedanything that could give you insight into how the clock was
working. Each line of the machine data shown above can be considered
a separate event, although its common for other machine data to have
events that span multiple or even hundreds of lines.
Splunk divides raw machine data into discrete pieces of information
known as events. When you do a simple search, Splunk retrieves the
events that match your search terms. Each event consists of discrete piec-
Exploring Splunk
14
es o dutu knovn us eds. ln cock dutu, the eds mght ncude $/!"#*,
3)#<%/, +"<=, *&8, 3"#%+, and 8/&=. If you think of groups of events orga-
nzed n u spreudsheet or dutubuse, the events ure the rovs und the eds
are the columns, as shown in Figure 2-1.
I|gute 21. C|ocl Iven|s |n a Spteadshee| Iotm
ln pructce, unother vuy to thnk o events s us u set o eds o keyvord/
value pairs. If represented as keyword/value pairs, the clock events look
like Figure 2-2.
I|gute 22. C|ocl Iven|s as I|e|ds o| Ieyvotd/Va|ue Ia|ts
Heres a real-world example of one of the most common and useful types
o muchne dutu. A veb server hus u og e thut records every LRL re-
quested from the server.
Some o the eds n veb server dutu ure:
!>)/#% ?@2 %)3/$%&3A2 +%%A 3/%+"*2 $%&%<$2 B8%/$2 =/C/==/=2
<$/= &D/#%
A visit to one webpage can invoke dozens of requests to retrieve text, im-
ages, and other resources. Each request is typically logged as a separate
event n u og e. 1he resut s u e thut ooks somethng ke lgure 2-3
(vthout the uncy hghghtng to hep you see the eds).
I|gute 2!. 1yp|ca| \eb Setvet Iog
Second
58
59
60
1
2
3
Minute
1
1
1
2
2
2
Hour
14
14
14
14
14
14
Day
23
23
23
23
23
23
Month
11
11
11
11
11
11
Year
2011
2011
2011
2011
2011
2011
Second=58, Minute=01, Hour=14, Day=23, Year=2011
Timestamp Http Command Browser Type IP Address
12.1.1.015 - - [01/Aug/2011:12:29:58 -0700] "GET /pages/hltabs_c.html HTTP/1.1" 200 1211 "http://webdev:2000/pages/" "Mozilla/5.0 AppleWebKit/102.1 (KHTML) Safari/102"
12.1.1.015 - - [01/Aug/2011:12:29:58 -0700] "GET /pages/joy.html HTTP/1.1" 200 0012 "http://webdev:2000/pages/" "Mozilla/5.0 AppleWebKit/102.1 (KHTML) Safari/102"
12.1.1.015 - - [01/Aug/2011:12:29:58 -0700] "GET /pages/dochomepage.html HTTP/1.1" 200 1000 "http://webdev:2000/pages/" "Mozilla/5.0 AppleWebKit/102.1 (KHTML) Safari/102"
Status Bytes Referrer
Chapter 2: Getting Data In
15
Types of Data Splunk Can Read
One of the common characteristics of machine data is that it almost al-
ways contains some indication of when the data was created or when an
event described by the data occurred. Given this characteristic, Splunks
indexes are optimized to retrieve events in time-series order. If the raw
data does not have an explicit timestamp, Splunk assigns the time at
which the event was indexed by Splunk to the events in the data or uses
other upproxmutons, such us the tme the e vus ust moded or the
timestamp of previous events.
The only other requirement is that the machine data be textual, not bi-
nury, dutu. lmuge und sound es ure common exumpes o bnury dutu
es. Some types o bnury es, ke the core dump produced vhen u
program crashes, can be converted to textual information, such as a stack
trace. Splunk can call your scripts to do that conversion before indexing
the data. Ultimately, though, Splunk data must have a textual representa-
tion to be indexed and searched.
Splunk Data Sources
During indexing, Splunk can read machine data from any number of
sources. The most common input sources are:
- Ics: Spunk cun montor specc es or drectores. l dutu s udded
to u e or u nev e s udded to u montored drectory, Spunk reuds
that data.
- thc nctwork: Splunk can listen on TCP or UDP ports, reading any
data sent.
- scriptcd inputs: Splunk can read the machine data output by pro-
grams or scripts, such as a Unix
command or a custom script that

monitors sensors.
Enough background: now lets get started working with Splunk.
Downloading, Installing, and Starting Splunk
We recommend that you install Splunk and add some machine data to
help you work through the topics discussed in this book. Everything well
cover can be done using Splunk Free (see below).
This section describes how to get Splunk up and running.
Exploring Splunk
16
Downloading Splunk
You can download fully functional Splunk for free, for learning or to sup-
port small to moderate use of Splunk. On the splunk.com home page,
you see this button:
Click it to begin downloading and installing Splunk on computers run-
ning Windows
, Mac, Linux
, and Unix.
Installing Splunk
Installing Splunk is easy, so well assume youll do that part on your own.
If you have any questions, refer to the Splunk Tutorial (http://splunk.com/
goto/book#tutorial), which covers everything in detail.
Starting Splunk
To start Splunk on Windows, launch the application from the Start menu.
Look for the Welcome screen, shown in Figure 2-4, and keep reading.
To start Splunk on Mac OS X or Unix, open a terminal window. Go to the
directory where you installed Splunk, go to the bin subdirectory and, at
the command prompt, type:
EF$A><#. $%&=%
The very last line of the information you see when Splunk starts is:
G+/ HA><#. (/B )#%/=C&!/ )$ &% +%%A-FF8"<=I3&!+)#/I
#&3/-9444
Follow that link to the login screen. If you dont have a username and
password, the default credentials are admin and changeme. After you log
in, the Welcome screen appears.
17
I|gute 24. 1he \e|come Scteen
The Welcome screen shows what you can do with your pristine instance
of Splunk: add data or launch the search app.
Bringing Data in for Indexing
The next step in learning and exploring Splunk is to add some data to the
index so you can explore it.
Were going to use some sample data for the purposes of this chapter.
You cun nd nstructons or gettng ths dutu here: http://splunk.com/goto/
book#add_data
There are two steps to the indexing process:
- Dovnoudng the sumpe e rom the Spunk vebste
- 1eng Spunk to ndex thut e
1o dovnoud the sumpe e, oov ths nk und suve the e to your
desktop: http://splunk.com/goto/book#sample_data
1o udd the e to Spunk:
1. From the Welcome screen, click Add Data.
2. Click from Ics and dircctorics on the bottom half of the screen.
3. Select Skip prcvicw.
4. Click the radio button next to UpIoad and indcx a Ic.
5. Seect the e you dovnouded to your desktop.
6. Click Savc.
Exploring Splunk
18
You're nshed uddng your dutu. Let's tuk ubout vhut Spunk s dong
behind the scenes.
Understanding How Splunk Indexes Data
Splunks core value to most organizations is its unique ability to index
machine data so that it can be quickly searched for analysis, reporting,
and alerts. The data that you start with is called raw data. Splunk indexes
raw data by creating a time-based map of the words in the data without
modifying the data itself.
Before Splunk can search massive amounts of data, it must index the
data. The Splunk index is similar to indexes in the back of textbooks,
vhch pont to puges vth specc keyvords. ln Spunk, the puges ure
called events.
I|gute 2S. 1he In|que Chatac|et|s||cs o| Sp|unl lndexes
Splunk divides a stream of machine data into individual events. Remem-
ber, un event n muchne dutu cun be us smpe us one ne n u og e or
as complicated as a stack trace containing several hundred lines.
Lvery event n Spunk hus ut eust the our deuut eds shovn n 1ube
2-1.
Indexing Pipeline Indexes
INDEX A
INDEX B
INDEX C
Search Head
The indexing pipeline reads
the machine data, divides it
into events, and identifes
some default felds
EVENT: raw text
+ fields such
as source,
sourcetype,
host, and _time
Machine data is
copied to the index
where it is available
during the search
process
The search head distributes
the search across many
indexes and consolidates
the results
splunk search command
Search
Results
Index
A
Merge
Results
Index
B
Index
C
12.1.1.140 - - [01/Aug/2009:09
:37:01 -0700] "GET /home/theme
s/ComBeta/images/btn_login.gif
12.1.1.140 - - [01/Aug
/2009:09:37:01 -0700]
"GET /home/themes/ComB
eta/images/btn_login.g
if HTTP/1.1" 304 - "ht
p://webdev:2000/home/i
ex.php" "Mozilla/5.0
.111 111..1 114 440 00 - - [01
///2 2000009 99: :009 99: :3337 77:01 -0
h th mm
1
pp: :/ // /w wwe e eeev vv 200 hom pp: :/ // /w w eev vvv:2200
eex hppp"" ""MM iil ll la/5 l
12.1.1.140 - - [01/Aug
/2009:09:37:01 -0700]
if HTTP/1.1" 304 - "ht
111 111..1 114 440 00 - - [01
99: :009 99: :3337 77:01 -0
h m th
1
wwe e eeev vv 200 hom w eev vvv:2200
hppp"" ""MM iil ll la/5 l
12.1.1.140 - - [01/Aug
/2009:09:37:01 -0700]
if HTTP/1.1" 304 - "ht
.111 111..1 114 440 00 - - [01
///2 200009 99: : 0 009 99: :3337 77:01 -0
h m th
1
pp: :/ // /w wwe e eeev vv 200 hom pp: :/ // /w w eev vvv:2200
eex hppp"" ""MM iil ll la/5 l
12.1
/2009 /2 ////
""""""""""G GGET
eeeeettta/i
iiiifff HT
pppp://w pppppp:://w
eeex.ph
.111
///2 20 00009 9 //
pppp: ::/ // /w w pppppp:::/ // //w :
eex ex e
Social
Media
Data
Credit
Card
Data
19
1ab|e 21. I|e|ds Sp|unl |vays lndexes
Field Answcrs thc qucstion fxampIcs
$"<=!/
Where did the data come from? es (FJ&=F>"DF), scripts
(38$!=)A%EB&%), net-
work feeds (KL@-05M)
$"<=!/%8A/
What kind of data is it?
&!!/$$N!"3B)#/*2
$8$>"D
+"$%
Which host or machine did the
data come from?
(/B$/=J/=452 !)$!"N
="<%/=
N%)3/
When did the event happen?
H&% O&= 75 46-5;-01
6456
1hese deuut eds ure ndexed uong vth the ruv dutu.
The timestamp (N%)3/) ed s specu becuuse Spunk ndexers uses t to
order events, enubng Spunk to ecenty retreve events vthn u tme
range.
Chapter 3 brings us to the place where most of the action happens:
Splunks search interface.
21
3 Searching with Splunk
Now that youve gained an understanding of the way Splunk indexes data
(in Chapter 2), it will be easier to understand what is happening when
you search with Splunk.
C course, the gou o seurch s to hep you nd exucty vhut you need. lt
cun meun terng, summurzng, und vsuuzng u urge umount o dutu,
to answer your questions about the data. At other times, you might need
to reguury expore urge umounts o dutu. Cten, you smpy vunt to nd
the needle in the haystack, the one buried event that threw everything off.
The Summary dashboard gives you a quick overview of the data visible
to you. Click launch scarch app on the Splunk WcIcomc tab. If youre
on the Splunk Homc tab, click Scarch under Your Apps. The Summary
dashboard displays, as shown in Figure 3-1
Exploring Splunk
22
I|gute !1. 1he Seatch apps Summaty dashboatd
Notice a few things about this dashboard:
- The scarch bar at the top is empty, ready for you to type in a search.
- The timc rangc pickcr to the right of the scarch bar permits time
range adjustment. You can see events from the last 15 minutes, for
example, or any desired time interval. For real-time streaming data,
you can select an interval to view, ranging from 30 seconds to an
hour.
- The AII indcxcd data panel displays a running total of the indexed
data.
Chapter 3: Searching With Splunk
23
The next three panels show the most recent or common values that have
been indexed in each category:
- The Sourccs pune shovs vhch es (or other sources) your dutu
came from.
- The Sourcc typcs panel shows the types of sources in your data.
- The Hosts panel shows which hosts your data came from.
Now, lets look at the Search navigation menus near the top of the page:
I|gute !2. Seatch nav|ga||on menus
- Summary is where we are.
- Scarch leads to the main search interface, the Scarch dashboard.
- Status lists dashboards on the status of your Splunk instance.
- Dashboards & Vicws lists your dashboards and views.
- Scarchcs & Rcports lists your saved searches and reports.
The next section introduces you to the Scarch dashboard.
The Search Dashboard
If you click the Search option or enter a search in the scarch bar, the
page switches to the Scarch dashboard (sometimes called the timeline or
ushtmene vev). \hen u seurch s kcked o, the resuts umost mme-
diately start displaying. For example, entering an asterisk (P) in the scarch
bar retrieves all the data in your default indexes, and a screen similar to
Figure 3-3 appears.
Exploring Splunk
24
I|gute !!. 1he Seatch dashboatd
Lets examine the contents of this dashboard:
- TimcIinc: A graphic representation of the number of events match-
ing your search over time.
- ficIds sidcbar: Reevunt eds uong vth event counts. 1hs menu
uso uovs you to udd u ed to the resuts.
- ficId discovcry switch: 1urns uutomutc ed dscovery on or o.
\hen Spunk executes u seurch und ed dscovery s on, Spunk
uttempts to denty eds uutomutcuy or the current seurch.
Results area
Fields
menu
Timeline
Timestamp Raw text
Field discovery switch
25
- RcsuIts arca: Shows the events from your search. Events are ordered
by Timcstamp, which appears to the left of each event. Beneath the
Raw tcxt o euch event ure uny eds seected rom the ficIds side-
bar for which the event has a value.
When you start typing in the scarch bar, context-sensitive information
appears below, with matching searches on the left and help on the right:
I|gute !4. He|p|u| |n|o appeats vhen you en|et |ex| |n |he seatch bat
Under the timc rangc pickcr, you see a row of icons:
I|gute !S. Seatch |cons
The scarch job controIs are only active when a search is running. If you
huven't run u seurch, or your seurch hus nshed, they ure nuctve und
greyed out. But if youre running a search that takes a long time to com-
plete, you can use these icons to control the search progress:
- Sending a search to the background lets it keep running to comple-
tion on the server while you run other searches or even close the
window and log out. When you click Scnd to background, the
scarch bar clears and you can continue with other tasks. When
Send to
background
Search job
controls
Pause
Finalize
Job inspector
Print results
Save search
Create menu
Cancel
Exploring Splunk
26
the ob s done, u notcuton uppeurs on your screen you're st
ogged n, othervse, Spunk emus you ( you've speced un emu
address). If you want to check on the job in the meantime, or at a
later time, click the ]obs link at the top of the page.
- Pausing a search temporarily stops it and lets you explore the re-
sults to that point. While the search is paused, the icon changes to a
play button. Clicking that button resumes the search from the point
where you paused it.
- Finalizing a search stops it before it completes, but retains the results
to that point and so you can view and explore it in the search view.
- In contrast, canceling a search stops it running, discards the results,
and clears them from the screen.
The ]ob inspcctor icon takes you to the ]ob inspcctor page, which shows
details about your search, such as the execution costs of your search,
debug messages, and search job properties.
Use the Savc menu to save the search, save the results, or save and share
the resuts. l you suve the seurch, you cun nd t on the Scarchcs & Rc-
ports menu. If you save the results, you can review them by clicking on
]obs in the upper right corner of the screen.
Use the Create menu to create dashboards, alerts, reports, event types,
and scheduled searches. Well explain those in detail in Chapter 5.
Moving down to the upper left corner of the RcsuIts area, you see the
following row of icons.
I|gute !6. Resu||s atea |cons
By default, Splunk shows events as a list, from most recent events to least,
but you can click on the Table icon to view your results as a table, or
you can click the Chart icon to view them as a chart. The Export button
exports your search results in various formats: CSV, raw events, XML, or
JSON.
List Table Chart Export search results
Options for displaying events
27
Events? Results? Whats the Difference?
1echn|ca||y speal|ng, te|t|eved even|s |tom yout |ndexes ate ca||ed even|s. l|
|hose even|s ate |tans|otmed ot summat|zed so |ha| |hete |s no |onget a one|o
one mapp|ng v||h even|s on d|sl, |hey ate ptopet|y ca||ed tesu||s. Iot examp|e,
a vebaccess even| te|t|eved |tom a seatch |s an even|, bu| |he |op IRI v|s||ed
today is a result. That said, we are not going to be that picky, and will use the two
|etms |n|etchangeab|y.
SPL: Search Processing Language
Splunk helps sift data from the mass of indexed events into a form that is
useful for answering real-world questions.
Figure 3-7 illustrates a common search pattern: retrieve events and gener-
ate a report. This search returns the top users in syslog errors.
I|gute !7. Hov a s|mp|e Sp|unl seatch |s ptocessed
The entire string
$"<=!/%8A/Q$8$>"D RSSTS U %"A <$/= U C)/>*$ I A/=!/#%
is called a search, and the pipe character (U) separates the individual
commands that make up the search.
Pipes
1he rst keyvord uter the ppe s the nume o the seurch commund. ln
this case the commands are %"A and C)/>*$. What command is retriev-
ing the events from the index? Well, there is an implied command called
Disk
Summarize into
table of top
ten users
Remove
percent
column
Final results
Events fetched
from disk
top user fields - percent
sourcetype = syslog ERROR | top user | fields - percent
sourcetype
syslog
syslog
other-source
syslog
syslog
syslog
other-source
syslog
other-source
<events>
raw
ERROR
ERROR
WARNING
WARNING
ERROR
IP address
user_A
user_A
user_A
user_A
user_B
<felds...>
User
user_01
user_02
...
user-10
count
22
17
5
percent
22
17
5
User
user_01
user_02
...
user-10
count
22
17
5
percent
22
17
5
User
user_01
user_02
...
user-10
count
22
17
5
Exploring Splunk
28
$/&=!+, at the beginning of any search that doesnt start with a pipe char-
acter. So, really, there are three search commands in the above search:
$/&=!+, %"A, and fields.
The results from each command are passed as input to the next com-
mand. If you have ever used a Linux shell such as bash, this concept is
probably familiar.
Implied AND
$"<=!/%8A/Q$8$>"D RSSTS tells the $/&=!+ command to retrieve only
events that have a $"<=!/%8A/ equal to $8$>"D AND contain the term
RSSTS.
top user
The next command, %"A, returns the most common values of the speci-
ed eds. y deuut, %"A returns the top 10 most common values for the
speced ed, n descendng order (thunk you, Duvd Lettermun). ln ths
cuse, the speced ed s <$/=, so %"A returns the users that appear most
often in syslog events that contain the term RSSTS. The output of %"A is a
table of 3 columns (<$/=, !"<#%, and A/=!/#%), with 10 rows of values.
Its also important to understand that the output of the %"A command be-
comes the input to the next command after the pipe. In this sense, %"A has
transformed the search results to a smaller set of values, which are further
rened by the next commund.
elds percent
The second command, C)/>*$, with an argument of V A/=!/#%, tells
Splunk to remove the A/=!/#% column from the output of the top com-
mand.
Exploratory Data Analysis: Spelunking with Splunk
\ha| || you don| lnov any|h|ng abou| |he da|a: Ce| ctea||ve and exp|ote. \ou
can do a seatch |ot P |o te|t|eve a|| even|s and |hen |eatn abou| |hem: |ool a|
some even|s, ex|tac| some |n|etes||ng |ool|ng |e|ds, ge| a %"A o| |ha| |e|d, see
hov |he even|s ate btolen up, pethaps det|ve some nev |e|ds based on o|het
|e|ds, c|us|et yout tesu||s, see hov one |e|d vat|es v||h ano|het |e|d, and so on.
(Iot mote ||ps abou| |eatn|ng vha|s |n a soutce |ha| you have |||||e lnov|edge
about, refer to http://splunk.com/goto/book#mining_tips.)
29
Before we dive into the search commands in Chapter 4, lets cover the
$/&=!+ command itself: a very special command that is critical for using
Splunk.
The search Command
The $/&=!+ command is the workhorse of Splunk. Its one of the simplest
and most powerful commands. Its such a basic command that you dont
even need to type t unyvhere beore the rst ppe, becuuse t s nvoked
implicitly at the head of a search, retrieving events from the indexes on
disk.
!o| a|| seatches te|t|eve da|a |tom Sp|unl |ndexes. Iot examp|e, |he )#A<%!$J
command teads da|a |tom a CSV ||e. 1o en|et commands o| |h|s sot| as |he |ts|
command, ptecede |hem v||h |he p|pe chatac|et. Iot examp|e: U )#A<%!$J
38C)>/E!$J
\hen t's not the rst commund n u seurch, the $/&=!+ command can
ter u set o resuts o the prevous seurch. 1o do ths, use the $/&=!+
command like any other commandwith a pipe character followed by
an explicit command name. For example, the command /=="= U %"A <=>
U $/&=!+ !"<#%WQ6 searches for events on disk that have the word /=="=,
nds the top LRLs, und ters uny LRLs thut ony occur once. ln other
words, of the 10 error events that %"A returns, show me only the ones
where there are two or more instances of that URL.
Table 3-1 shows a few examples of implicit calls to the $/&=!+ command
and their results.
1ab|e !1. lmp||c|| seatch commands
Scarch Argumcnts RcsuIt
X(&=# TS /=="=Y ZTG C&)>P
Retrieves all events containing either
warn or error, but not those that
have fail, fails, failed, failure,
etc.
[*&%&B&$/ /=="=\ C&%&> *)$.
Retrieves all events containing the
phrase database error, fatal, and
disk (the AND is implied).
+"$%Q3&)#N(/BN$/=J/= */>&8W6
Retrieves all events that have a host
ed vth u vuue o mun_veb_serv-
er und u deuy ed vth u vuue
greater than 2.
Exploring Splunk
30

Tips for Using the search Command
Here are a few tips for using the $/&=!+ command. They apply to many
other commands as well.
Case-sensitivity
Keyword arguments to the $/&=!+ command are not case-sensitive, but
ed numes ure. (See Appendx or more detus ubout cuse-senstvty.)
Using quotation marks in a search
You need quotuton murks uround phruses or ed vuues thut contun
breaking characters such as whitespace, commas, pipes, square brackets,
and equals signs. So, +"$%Q(/B4: s ne, but the host vuue hus spuces,
for example, youll need quotes around the value, as in +"$%Q\(/B$/=J/=
]:\. In addition, to search for reserved keywords (e.g., AND, OR, NOT,
etc.), use quotes.
To search for quotes use a backslash to escape the quote character. To
nd the phruseSp|unl changed |||e ||se|| |ot meyoud search for:
[HA><#. !+&#D/* ^\>)C/ )%$/>C^\ C"= 3/\
Boolean logic
Argumentskeyvords und edsto the $/&=!+ command are ANDed
together, implicitly.
You can specify that either one of two or more arguments should be true
using the OR keyword, in uppercase. OR has higher precedence than
AND, so you can think of arguments using OR as having parentheses
around them.
1o ter out events thut contun u purtcuur vord, use the NC1 keyvord.
Finally, you can use parentheses explicitly to make things more clear if
you want to. For example, a search for _ 8 TS ` ZTG ( is the same as _
,ZL X8 TS `Y ,ZL ZTG (.
Subsearches
The $/&=!+ command, like all commands, can be used as a subsearcha
search whose results are used as an argument to another search com-
mund. Subseurches ure encosed n squure bruckets. lor exumpe, to nd
all syslog events from the user that had the last login error, use the follow-
ing command:
$"<=!/%8A/Q$8$>"D a$/&=!+ >"D)# /=="= U =/%<=# <$/=b
31
Here, a search for events having the terms >"D)# and /=="= is performed,
returnng the rst <$/= value found, say B"B, followed by a search for
$"<=!/%8A/Q$8$>"D <$/=QB"B.
If youre ready to continue your adventure in learning Splunk, Chapter
4 ntroduces you to more communds you v nd mmedutey hepu.
33
4 SPL: Search Processing
Language
In Chapter 3, we covered the most basic Splunk command in the SPL:
search. This chapter describes some of the other SPL commands youll
want to learn.
This chapter takes a bare bones, learn-by-example approach to SPL com-
mands. For complete reference documentation, see http://docs.splunk.
com.
Table 4-1 summarizes the SPL commands covered in this chapter, by
category.
1ab|e 41. Common SII Commands
Catcgory Dcscription Commands
Sorting RcsuIts Ordering results and (optionally)
limiting the number of results.
$"=%
fiItcring RcsuIts Taking a set of events or results and
terng them nto u smuer set o
results.
$/&=!+
(+/=/
*/*<A
+/&*
%&)>
Grouping RcsuIts Grouping events so you can see pat-
terns.
%=&#$&!%)"#
Rcporting RcsuIts Taking search results and generating
a summary for reporting.
%"AF=&=/
$%&%$
!+&=%
%)3/!+&=%
fiItcring, Modify-
ing, and Adding
ficIds
lterng out (removng) some eds
to focus on the ones you need, or
modyng or uddng eds to enrch
your results or events.
C)/>*$
=/A>&!/
/J&>
=/_
>"".<A
Sorting Results
Sorting results is the province of the (you guessed it!) $"=% command.
sort
The $"=% commund sorts seurch resuts by the speced eds.
Table 4-2 shows some examples.
Exploring Splunk
34
Shorthand for Part of a Search
l| ve shov jus| pat| o| a set|es o| commands (as ve do |n 1ab|e 42), you|| see:
EEE U
1h|s means |ha| some seatch pteceded |h|s command, bu| ve ate |ocus|ng on
vha| comes a||etvatd.
1ab|e 42. $"=% Command Ixamp|es
Command RcsuIt
c U $"=% 4 C)/>*5
Sort results in ascending order B8
C)/>*5, returning all results (4 means
return them all; dont stop at 10,000,
which is the default).
c U $"=% C)/>*52IC)/>*6
Sort results by C)/>*5 in ascending
order, and then by C)/>*6 in descend-
ing order, returning up to 10,000 results
(the default).
c U $"=% 544 VC)/>*52dC)/>*6
Sort results in descending order by
C)/>*5, and then in ascending order by
C)/>*6, returnng the rst l00 sorted
results.
c U $"=% C)>/#&3/
c U $"=% #<3XC)>/#&3/Y
c U $"=% $%=XC)>/#&3/Y
Sort results by C)>/#&3/:
- 1he rst commund ets Spunk
decde hov to sort the ed vuues.
- The second command tells Splunk
to sort the values numerically.
- The third command tells Splunk to
sort the values lexicographically.
Hint: Ascending order is the default for search results. To reverse the order of
tesu||s, use a m|nus s|gn |n |ton| o| a |e|d used |o otdet |he tesu||s.
Figure 4-1 illustrates the second example. Well sort by ascending prices
und descendng rutngs. 1he rst resut s the cheupest tem vth the hgh-
est user rating.
Chapter 4: SPL: Search Processing Language
35
sort by price in
ascending order
sort by rating in
descending order
...| sort price,-rating
previous
search results
price
9.99
9.88
22.50
22.50
48.88
9.99
9.99
48.88
22.50
rating
1
2
2
3
3
4
4
5
5
price
9.88
9.99
9.99
9.99
22.50
22.50
22.50
48.88
48.88
rating
2
1
4
4
2
3
5
3
5
price
9.88
9.99
9.99
9.99
22.50
22.50
22.50
48.88
48.88
rating
2
4
4
1
5
3
2
5
3
felds
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
felds
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
felds
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
I|gute 41. $"=% Command
Filtering Results
These commands take search results from a previous command and re-
duce them to a smaller set of results. In other words, youre narrowing
down your view of the data to show only the results you are looking for.
where
The (+/=/ terng commund evuuutes un expresson or terng resuts.
If the evaluation is successful and the result is GSKR, the result is retained;
otherwise, the result is discarded. For example:
$"<=!/Qe"BN>)$%)#D$ U (+/=/ $&>&=8 W )#*<$%=8N&J/=&D/
This example retrieves jobs listings and discards those whose salary is not
greater than the industry average. It also discards events that are missing
either the $&>&=8 ed or the )#*<$%=8N&J/=&D/ ed.
1hs exumpe compures tvo eds$&>&=8 and )#*<$%=8N&J/=&D/
something we can only do with the (+/=/ command. When comparing
ed vuues to teru vuues, smpy use the $/&=!+ command:
$"<=!/Qe"BN>)$%)#D$ $&>&=8W94444
Exploring Splunk
36
1ab|e 4!. (+/=/ Command Ixamp|es
Command RcsuIt
c U (+/=/ *)$%&#!/F%)3/ W 544
Keep results whose *)$%&#!/
ed vuue dvded by the %)3/
ed vuue s greuter thun l00.
c U (+/=/ >)./X$=!2 [54E:E5;0Ef\Y
TS !)*=3&%!+X[54E:E5;0E4F60\2
*$%Y
Keep results that match the IP
uddress or ure n the speced
subnet.
Figure 4-2 illustrates the command where *)$%&#!/F%)3/ W 544.
I|gute 42. (+/=/ Command Ixamp|e
Tips for Using where
Like the /J&> command, the (+/=/ command works with a large set of
expression evaluation functions (see Appendix E for a complete list).
dedup
Removing redundant data is the point of the */*<A terng commund.
1hs commund removes subsequent resuts thut mutch speced crteru.
1hut s, ths commund keeps ony the rst !"<#% results for each combina-
ton o vuues o the speced eds. l !"<#% s not speced, t deuuts
to l und returns the rst resut ound (vhch s usuuy the most recent).
fnal results
...| where distance/time > 100
evaluate (distance/time>100)
and keep only events for
which the result is TRUE
previous
search results
distance
50
100
200
300
300
100
500
time
10
10
5
3
2
0.5
2
distance
50
100
200
300
300
100
500
time
10
10
5
3
2
0.5
2
distance/time>100
. . . . . . . . . . . . . . . . . . . FALSE
. . . . . . . . . . . . . . . . . . . FALSE
. . . . . . . . . . . . . . . . . . . FALSE
. . . . . . . . . . . . . . . . . . . FALSE
. . . . . . . . . . . . . . . . . . . .TRUE
. . . . . . . . . . . . . . . . . . . .TRUE
. . . . . . . . . . . . . . . . . . . .TRUE
distance
300
100
500
time
2
0.5
2
37
1ab|e 44. */*<A Command Ixamp|es
Command Result
*/*<A +"$%
leep the rst resut or euch unque
+"$%.
*/*<A 7 $"<=!/
leep the rst three resuts or euch
unique $"<=!/.
*/*<A $"<=!/ $"=%B8 I*/>&8
$"<=!/ uter rst sortng the resuts
by the */>&8 ed n descendng
order. Effectively this keeps the result
with the largest delay value for each
unique $"<=!/.
*/*<A 7 $"<=!/2+"$%
leep the rst three resuts or euch
unique combination of $"<=!/ and
+"$% values.
*/*<A $"<=!/ .//A/3A%8Q%=</
$"<=!/, also keeping those with no
$"<=!/ ed.
Figure 4-3 illustrates the command */*<A 7 $"<=!/.
I|gute 4!. */*<A Command Ixamp|e
Key Points
- To keep all results but remove duplicate values, use the .//A/J/#%$
option.
- 1he resuts returned ure the rst resuts ound vth the combnuton
o speced ed vuuesgeneruy the most recent ones. Lse the
$"=%B8 clause to change the sort order if needed.
previous
search results
fnal results
...| dedup 3 source
for events with matching
source feld values, remove
all except the frst three
source
source_A
source_A
source_B
source_B
source_A
source_A
source_A
source_B
source_B
feld2
f2_value1
f2_v alue2
f2_v alue3
f2_v alue4
f2_v alue5
f2_v alue6
f2_v alue7
f2_v alue8
f2_v alue9
<felds. . . >
. . ..
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
source
source_A
source_A
source_B
source_B
source_A
source_A
source_A
source_B
source_B
feld2
f2_value1
f2_v alue2
f2_v alue3
f2_v alue4
f2_v alue5
f2_v alue6
f2_v alue7
f2_v alue8
f2_v alue9
<felds. . . >
. . ..
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
source
source_A
source_A
source_B
source_B
source_A
source_B
feld2
f2_value1
f2_v alue2
f2_v alue3
f2_v alue4
f2_v alue5
f2_v alue8
<felds. . . >
. . ..
. . .
. . .
. . .
. . .
. . .
Exploring Splunk
38
- leds vhere the speced eds do not u exst ure retuned by de-
fault. Use the .//A#<>>Qg%=</FC&>$/W option to override the default
behavior, if desired.
head
The +/&* terng commund returns the rst count resuts. Lsng +/&* per-
mts u seurch to stop retrevng events rom dsk vhen t nds the desred
number of results.
Heads or Tails?
The opposite of the +/&* command |s |he %&)> command, vh|ch te|utns |he
|as| tesu||s, ta|het |han |he |ts|. 1he tesu||s ate te|utned |n tevetse otdet, s|at||ng
a| |he end o| |he tesu||s. Ieep |n m|nd |ha| |ts| |s te|a||ve |o |he |npu| otdet o|
even|s, vh|ch |s usua||y |n descend|ng ||me otdet, mean|ng |ha|, |ot examp|e,
+/&* 54 returns the latest 10 events.
1ab|e 4S. +/&* Command Ixamp|es
Command RcsuIt
c U +/&* 0
Return the rst 5 resuts.
c U +/&* X&!%)"#Q\$%&=%<A\Y
Return the rst events unt ve reuch
an event that does NOT have an &!I
%)"# ed vth the vuue $%&=%<A.
1he rst exumpe n 1ube 4-5, +/&* 0, is illustrated in Figure 4-4.
I|gute 44. +/&* Command Ixamp|e
...| head 5
retrieve only the frst fve (5) search operator
feld1
1
2
3
4
5
6
7
8
9
<felds. . . >
. . ..
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
feld1
1
2
3
4
5
<felds. . . >
. . ..
. . .
. . .
. . .
. . .
39
Grouping Results
The %=&#$&!%)"# command groups related events.
transaction
The %=&#$&!%)"# command groups events that meet various constraints
into transactionscollections of events, possibly from multiple sources.
Lvents ure grouped together u trunsucton denton construnts ure
met. Transactions are composed of the raw text (the N=&( ed) o euch
member event, the timestamp (the N%)3/ ed) o the eurest member
event, the unon o u other eds o euch member event, und some ud-
dtonu eds the descrbe the trunsucton such us *<=&%)"# and /J/#%I
!"<#%.
1ab|e 46. %=&#$&!%)"# Command Ixamp|es
Command RcsuIt
c U %=&#$&!%)"# !>)/#%)A
3&_A&<$/Q0$
Group events that share the same client IP
address and have no gaps or pauses longer
thun ve seconds.
With this command, the search results may
have multiple values for the +"$% ed. lor
example, requests from a single IP address
could come from multiple hosts if multiple
people are accessing the server from the
same location.
c U %=&#$&!%)"# !>)/#I
%)A +"$% 3&_$A&#Q74$
3&_A&<$/Q0$
Group events that share the same unique
combination of client IP address and host,
vhere the rst und ust events ure no more
than 30 seconds apart and no event in the
trunsucton occurred no more thun ve
seconds apart.
ln contrust vth the rst exumpe, euch
result event has a distinct combination of
the IP address (!>)/#%)A) and host value
within the limits of the time constraints.
Therefore, you should not see different val-
ues of +"$% or !>)/#%)A addresses among
the events in a single transaction.
Exploring Splunk
40
$"<=!/%8A/Q&!!/$$P
&!%)"#QA<=!+&$/ U
%=&#$&!%)"# !>)/#%)A
3&_$A&#Q543 3&_/J/#%$Q7
Retrieve web access events that have an
&!%)"#QA<=!+&$/ value. These events are
then grouped by the %=&#$&!%)"# com-
mand if they share the same !>)/#%)A,
where each session lasts no longer than 10
minutes and includes no more than three
events.
c U %=&#$&!%)"# hHRHI
H?TZ?L !>)/#%)A
$%&=%$()%+Q\$)D#"#\
/#*$()%+Q\A<=!+&$/\ U
(+/=/ *<=&%)"#WQ5
Group events together that have the same
session ID (hHRHH?TZ?L) and come from
the same IP address (!>)/#%)A) and where
the rst event contuns the strng, [$)D#"#\
and the last event contains the string, [A<=I
!+&$/E\
1he seurch denes the rst event
in the transaction as events that in-
clude the string, [$)D#"#\, using the
$%&=%$()%+Q\$)D#"#\ argument. The
/#*$()%+Q\A<=!+&$/\ argument does the
same for the last event in the transaction.
This example then pipes the transactions
into the (+/=/ command, which uses the
*<=&%)"# ed to ter out trunsuctons thut
took less than a second to complete.
The second example in Table 4-6, %=&#$&!%)"# !>)/#%)A 3&_$A&#Q74$
3&_A&<$/Q0$, is illustrated in Figure 4-5.
I|gute 4S. %=&#$&!%)"# Command Ixamp|e
41
Key Points
All the %=&#$&!%)"# command arguments are optional, but some con-
strunts must be speced to dene hov events ure grouped nto trunsuc-
tions.
Spunk does not necessury nterpret the trunsucton dened by mutpe
eds us u conuncton (C)/>*5 ,ZL C)/>*6 ,ZL C)/>*7) or a disjunc-
tion (C)/>*5 TS C)/>*6 TS C)/>*7) o those eds. l there s u trunstve
reutonshp betveen the eds n the gC)/>*$ >)$%W, the %=&#$&!%)"#
command uses it.
For example, if you searched for %=&#$&!%)"# +"$% !"".)/, you might
see the following events grouped into a single transaction:
/J/#%Q5 +"$%Q&
/J/#%Q6 +"$%Q& !"".)/QB
/J/#%Q7 !"".)/QB
1he rst tvo events ure oned becuuse they huve +"$%Q& in common and
then the third is joined with them because it has !"".)/QB in common
with the second event.
1he trunsucton commund produces tvo eds:
- *<=&%)"#: derence betveen the tmestumps or the rst und ust
events in the transaction.
- /J/#%!"<#%: number of events in the transaction.
Although the $%&%$ command (covered later in this section) and the
%=&#$&!%)"# command both enable you to aggregate events, there is an
important distinction:
- $%&%$ calculates statistical values on events grouped by the value of
eds (und then the events ure dscurded).
- %=&#$&!%)"# groups events, and supports more options on how they
ure grouped und retuns the ruv event text und other ed vuues
from the original events.
Reporting Results
Reporting commands covered in this section include %"A, $%&%$, !+&=%,
and %)3/!+&=%.
top
Cven u st o eds, the %"A command returns the most frequently oc-
currng tupe o those ed vuues, uong vth ther count und percent-
Exploring Splunk
42
uge. l you specy un optonu by-cuuse o uddtonu eds, the most
requent vuues or euch dstnct group o vuues o the by-cuuse eds
are returned.
The opposite of top is rare
The opposite of the %"A command |s |he =&=/ command. Some||mes you van| |o
lnov vha| |s |he |eas| common va|ue |ot a |e|d (|ns|ead o| |he mos| common).
The =&=/ command does exac||y |ha|.
1ab|e 47. %"A Command Ixamp|es
Command RcsuIt
c U %"A 64 <=>
Return the 20 most common URLs.
c U %"A 6 <$/= B8 +"$%
Return the top 2 <$/= values for each
host.
c U %"A <$/=2 +"$%
Return the top 10 (default) <$/=I
+"$% combinations.
The second example in Table 4-7, %"A 6 <$/= B8 +"$%, is illustrated in
Figure 4-6.
I|gute 46. %"A Command Ixamp|e
intermediate results:
identifying count &
percent of user values
for each host value
...|
top 2 user by host
previous
search results
fnal results
host
host-1
host-1
host-1
host-1
host-1
host-1
host-1
host-2
host-2
host-2
host-2
host-2
user
user_A
user_A
user_B
user_C
user_C
user_C
user_D
user_E
user_E
user_F
user_G
user_G
<felds. . . >
. . ..
. . ..
. . ..
. . ..
. . ..
. . ..
. . ..
. . ...
. . ..
. . ..
. . ..
. . ..
host
host-1
host-1
host-1
host-1
host-2
host-2
host-2
user
user_A
user_B
user_C
user_D
user_E
user_F
user_G
count
2
1
3
1
2
1
2
percent
16.67
8.33
25.00
8.33
16.67
8.33
16.67
host
host-1
host-1
host-2
host-2
user
user_C
user_A
user_E
user_G
count
3
2
2
2
percent
25.00
16.67
16.67
16.67
43
stats
The $%&%$ command calculates aggregate statistics over a dataset, similar
to SQL aggregation. The resultant tabulation can contain one row, which
represents the aggregation over the entire incoming result set, or a row for
euch dstnct vuue o u speced by-cuuse.
Theres more than one command for statistical calculations. The $%&%$,
!+&=%, and %)3/!+&=% commands perform the same statistical calcula-
tions on your data, but return slightly different result sets to enable you to
more easily use the results as needed.
- The $%&%$ command returns a table of results where each row rep-
resents a single unique combination of the values of the group-by
eds.
- The !+&=% command returns the same table of results, with rows as
uny urbtrury ed.
- The %)3/!+&=% command returns the same tabulated results, but the
rov s set to the nternu ed, N%)3/, which enables you to chart
your results over a time range.
Table 4-8 shows a few examples of using the $%&%$ command.
What as means
Note: 1he use o| |he leyvotd as |n some o| |he commands |n 1ab|e 414. &$ is
used |o tename a |e|d. Iot examp|e, $<3XA=)!/Y as [S/J/#</\ means add up a||
the A=)!/ |e|ds and name |he co|umn shov|ng |he tesu||s Revenue.
1ab|e 48. $%&%$ Command Ixamp|es
Command RcsuIt
c U $%&%$ *!X+"$%Y
Return the distinct count (i.e.,
unique) of +"$% values.
c U $%&%$ &JDX.BA$Y B8 +"$%
Return the average transfer rate for
each host.
c U $%&%$
!"<#%X/J&>X3/%+"*Q\iRG\YY
&$ iRG2
!"<#%X/J&>X3/%+"*Q\@THG\YY &$
@THG B8 +"$%
Return the number of different types
of requests for each Web server
(+"$%). The resultant table contains
a row for each host and columns for
the GET and POST request method
counts.
EEE U %"A >)3)%Q544 =/C/=/=N
*"3&)# U $%&%$ $<3X!"<#%Y &$
%"%&>
Return the total number of hits from
the top 100 values of =/C/=/=N*"I
3&)#.
Exploring Splunk
44
c U $%&%$ !"<#%2
3&_XO&D#)%<*/Y2
3)#XO&D#)%<*/Y2
=&#D/XO&D#)%<*/Y2
&JDXO&D#)%<*/Y B8 S/D)"#
Using USGS Earthquakes data, return
the number of quakes and additional
statistics, for each S/D)"#.
c U $%&%$ J&></$XA="*<!%N%8A/Y
&$ G8A/2 J&></$XA="*<!%N#&3/Y
&$ Z&3/2 $<3XA=)!/Y &$ [S/JI
/#</\ B8 A="*<!%N)* U =/I
#&3/ A="*<!%N)* &$ [@="*I
<!% ?L\ U /J&> S/J/#</Q\j
[E%"$%=)#DXS/J/#</2\!"33&$\Y
Return a table with G8A/, Z&3/, and
S/J/#</ columns for each A="*I
<!%N)* sold at a shop. Also, format
the S/J/#</ as $123,456.
The third example in Table 4-8, retrieving the number of GET and POST
requests per host, is illustrated in Figure 4-7.
I|gute 47. $%&%$ Command Ixamp|e
Table 4-9 lists statistical functions that you can use with the $%&%$ com-
mand. (These functions can also be used with the !+&=% and %)3/!+&=%
commands, which are discussed later.)
1ab|e 40. $%&%$ S|a||s||c a| Iunc||ons
MathcmaticaI CaIcuIations
&JDXkY
Returns uveruge o the vuues o ed X, see uso,
3/&#XkY.
!"<#%XkY
Returns the number o occurrences o the ed X, to
ndcute u ed vuue to mutch, ormut the X urgument
as an expression: /J&>XC)/>*QlJ&></lY.
*!XkY
Returns the count o dstnct vuues o ed X.
3&_XkY
Returns the muxmum vuue o ed X. l the vuues ure
non-numeric, the max is determined per lexicographic
ordering.
3/*)&#XkY
Returns the mdde-most vuue o ed X.
3)#XkY
Returns the mnmum vuue o ed X. l the vuues ure
non-numeric, the min is determined per lexicographic
ordering.
45
3"*/XkY
Returns the most requent vuue o ed X.
A/=!gA/=!/#%I
#<3WXkY
Returns the gA/=!/#%I#<3W-th vuue o ed X, or
example, A/=!0X%"%&>Y returns the 5
th
percentile value
of the %"%&> ed.
=&#D/XkY
Returns the difference between the max and min values
o ed X, provded vuues ure numerc.
$%*/JXkY
Returns the sumpe stundurd devuton o ed X. You
cun use vdcurds vhen you specy the ed nume,
for example, "*delay", which matches both "delay" and
"xdelay".
$<3XkY
Returns the sum o the vuues o ed X.
J&=XkY
Returns the sumpe vurunce o ed X.
VaIuc ScIcctions
C)=$%XkY
Returns the rst vuue o ed X, opposte o >&$%XkY.
>&$%XkY
Returns the ust vuue o ed X, opposte o C)=$%XkY.
Ceneruy, u ed's ust vuue s the most chronoogcuy
oldest value.
>)$%XkY
Returns the st o u vuues o ed X us u mutvuue
entry. The order of the values matches the order of input
events.
J&></$XkY
Returns a list (as a multivalue entry) of all distinct values
o ed X, ordered excogruphcuy.
timcchart onIy (not appIicabIc to chart or stats)
A/=N*&8XkY
Returns the rute o ed X per duy
A/=N+"<=XkY
Returns the rute o ed X per hour
A/=N3)#<%/XkY
Returns the rute o ed X per mnute
A/=N$/!"#*XkY
Returns the rute o ed X per yeur
Notc: AII functions cxccpt thosc in thc timcchart onIy catcgory arc ap-
pIicabIc to thc !+&=%, $%&%$, and %)3/!+&=% commands.
chart
The !+&=% command creates tabular data output suitable for charting. You
specify the x-axis variable using "J/= or B8.
Table 4-10 shows a few simple examples of using the !+&=% command;
for more realistic scenarios, see Chapter 6.
Exploring Splunk
46
1ab|e 416. !+&=% Command Ixamp|es
Command RcsuIt
c U !+&=% 3&_X*/>&8Y "J/= +"$%
Return 3&_X*/>&8) for each value of
+"$%.
c U !+&=% 3&_X*/>&8Y B8 $)`/
B)#$Q54
Chart the maximum */>&8 by $)`/,
where $)`/ is broken down into a
maximum of 10 equal-size buckets.
c U !+&=% /J&>X&JDX$)`/YF
3&_X*/>&8YY &$ =&%)" B8 +"$%
<$/=
Chart the ratio of the average (mean)
$)`/ to the maximum delay for each
distinct +"$% and <$/= pair.
EEE U !+&=% *!X!>)/#%)AY "J/=
*&%/N+"<= B8 !&%/D"=8N)*
<$/#<>>QC
Chart the number of unique !>)I
/#%)A values per hour by category.
<$/#<>>QC excudes eds thut don't
have a value.
c U !+&=% !"<#% "J/= O&D#)%<*/
B8 S/D)"# <$/"%+/=QC
Chart the number of earthquakes by
O&D#)%<*/ and S/D)"#. Use the
<$/"%+/=QC argument to not output
an other value for rarer S/D)"#$.
c U !+&=%
!"<#%X/J&>X3/%+"*Q\iRG\YY
&$ iRG2
!"<#%X/J&>X3/%+"*Q\@THG\YY &$
@THG B8 +"$%
Chart the number of iRG and @THG
page requests that occurred for each
Web server (+"$%)
Figures 4-8 (tabulated results) and 4-9 (bar chart on a logarithmic scale)
illustrate the results of running the last example in Table 4-10:
I|gute 48. !+&=% Command Ixamp|e1abu|a|ed Resu||s
47
I|gute 40. !+&=% Command Ixamp|eRepot| Bu||det Iotma||ed Chat|
timechart
The %)3/!+&=% command creates a chart for a statistical aggregation ap-
ped to u ed ugunst tme us the x-uxs.
Table 4-11 shows a few simple examples of using the %)3/!+&=% com-
mand. Chapter 6 offers more examples of using this command in context.
1ab|e 411. %)3/!+&=% Command Ixamp|e
Command RcsuIt
c U %)3/!+&=% $A&#Q53 &JDXm@KY
B8 +"$%
Chart the average value of CPU usage
each minute for each host.
c U %)3/!+&=% $A&#Q5* !"<#%
B8 A="*<!%I%8A/
Chart the number of purchases made
daily for each type of product. The
$A&#Q5* argument buckets the count
of purchases over the week into daily
chunks.
cU %)3/!+&=% &JDX!A<N$/!"#*$Y
B8 +"$% U "<%>)/=
Chart the average !A<N$/!"#*$ by
+"$% and remove outlying values that
may distort the timecharts y-axis.
cU %)3/!+&=% A/=N+"<=XA=)!/Y
B8 A="*<!%N#&3/
Chart hourly revenue for the prod-
ucts that were purchased yesterday.
The A/=N+"<=XY function sums the
vuues o the prce ed or euch tem
XA="*<!%N#&3/Y and scales that
sum appropriately depending on the
timespan of each bucket.
Exploring Splunk
48
c U %)3/!+&=%
!"<#%X/J&>X3/%+"*Q\iRG\YY
&$ iRG2
!"<#%X/J&>X3/%+"*Q\@THG\YY &$
@THG
Chart the number of page requests
over time. The !"<#%XY function and
/J&> expressions are used to count
the different page request methods,
iRG and @THG.
c U %)3/!+&=% A/=N
+"<=X/J&>X3/%+"*Q\iRG\YY
&$ n)/($2 A/=N
+"<=X/J&>X&!%)"#Q\A<=!+&$/\YY
&$ @<=!+&$/$
For an ecommerce website, chart
A/=N+"<= the number of produc t
views and purchasesanswering
the question, how many views did
not lead to purchases?
The fourth example in Table 4-11, charting hourly revenues by product
nume, s ustruted n gures 4-l0 und 4-ll.
I|gute 416. %)3/!+&=% Command Ixamp|e1abu|a|ed Resu||s
I|gute 411. %)3/!+&=% Command Ixamp|eIotma||ed 1|mechat|
Filtering, Modifying, and Adding Fields
1hese communds hep you get ony the desred eds n your seurch re-
sults. You might want to simplify your results by using the C)/>*$ com-
mund to remove some eds. You mght vunt to muke your ed vuues
49
more readable for a particular audience by using the =/A>&!/ command.
Cr you mght need to udd nev eds vth the hep o communds such us
/J&>, =/_, and >"".<A:
- The /J&> commund cucuutes the vuue o u nev ed bused on
other eds, vhether numercuy, by concutenuton, or through
Boolean logic.
- The =/_ commund cun be used to creute nev eds by usng reguur
expressons to extructng putterned dutu n other eds.
- The >"".<A commund udds eds bused on ookng ut the vuue n
un event, reerencng u ookup tube, und uddng the eds n mutch-
ing rows in the lookup table to your event.
1hese communds cun be used to creute nev eds or they cun be used to
overvrte the vuues o exstng eds. lt's up to you.
elds
The C)/>*$ commund removes eds rom seurch resuts. 1ypcu com-
mands are shown in Table 4-6.
1ab|e 412. C)/>*$ Command Ixamp|es
Command RcsuIt
c U C)/>*$ V C)/>*52 C)/>*6
Remove C)/>*5 and C)/>*6 from
the search results.
c U C)/>*$ C)/>*5 C)/>*6
Keep only C)/>*5 and C)/>*6.
c U C)/>*$ C)/>*5 /=="=P
Keep only C)/>*5 und u eds
whose names begin with /=="=.
c U C)/>*$ C)/>*5 C)/>*6 U
C)/>*$ I NP
Keep only C)/>*5 and C)/>*6 and
remove u nternu eds (vhch
begin with an underscore). (Note:
Removng nternu eds cun cuuse
Splunk Web to render results in-
correctly and create other search
problems.)
1he rst exumpe n 1ube 4-l2, C)/>*$ V C)/>*5, C)/>*6, is illustrated
in Figure 4-12.
Exploring Splunk
50
I|gute 412. C)/>*$ Command Ixamp|e
Key Points
lnternu eds, .e., eds vhose numes sturt vth un underscore, ure unu-
fected by the C)/>*$ commund, uness expcty speced.
replace
The =/A>&!/ commund perorms u seurch-und-repuce o speced ed
values with replacement values.
The values in a search and replace are case-sensitive.
1ab|e 41!. =/A>&!/ Command Ixamp|es
Command RcsuIt
=/A>&!/ P>"!&>+"$% ()%+ >"!I
&>+"$% )# +"$%
Change any +"$% value that ends
with >"!&>+"$% to >"!&>+"$%.
=/A>&!/ 4 ()%+ m=)%)!&> 2 5
()%+ R=="= )# 3$DN>/J/>
Change 3$DN>/J/> values of 4 to
m=)%)!&>, and change 3$DN>/J/>
values of 5 to R=="=.
=/A>&!/ &<D ()%+ ,<D<$% )#
$%&=%N3"#%+ /#*N3"#%+
Change any $%&=%N3"#%+ or /#*N
3"#%+ value of &<D to ,<D<$%.
=/A>&!/ 561E4E4E5 ()%+ >"!&>I
+"$%
Chunge u ed vuues o 561E4E4E5
to >"!&>+"$%.
The second example in Table 4-13, =/A>&!/ 4 ()%+ m=)%)!&> 2 5 with
Lrror n msg_eve, s ustruted n lgure 4-l3.
previous
search results
fnal results
...|
fields - field1 field2
remove feld1 and feld 2
columns
feld1
1
2
3
4
5
6
7
8
9
feld1
A
B
C
D
E
F
G
H
I
feld1
a
b
c
d
e
f
g
h
i
<felds. . . >
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
feld1
a
b
c
d
e
f
g
h
i
<felds. . . >
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
feld1
1
2
3
4
5
6
7
8
9
feld1
A
B
C
D
E
F
G
H
I
feld1
a
b
c
d
e
f
g
h
i
<felds. . . >
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
51
I|gute 41!. =/A>&!/ Command Ixamp|e
eval
The /J&> command calculates an expression and puts the resulting value
nto u nev ed. 1he /J&> and (+/=/ commands use the same expression
syntax; Appendix E lists all the available functions.
1ab|e 414. /J&> Command Ixamp|es
Command RcsuIt
c U /J&> J/>"!)%8Q*)$%&#!/F
%)3/
Set J/>"!)%8 to distance divided by
time.
c U /J&> $%&%<$ Q )CX/=="= QQ
6442 [To\2 [R=="=\Y
Set status to OK if error is 200; other-
wise set status to Error.
c U /J&> $<3N"CN&=/&$ Q A)XY
P A"(X=&*)<$N&2 6Y d A)XY P
A"(X=&*)<$NB2 6Y
Set sum_o_ureus to be the sum o the
areas of two circles.
lgure 4-l4 ustrutes the rst exumpe n 1ube 4-l4, /J&>
J/>"!)%8Q*)$%&#!/F%)3/.
previous
search results
fnal results
...|
replace 0 with Critical,
1 with Error in msg_level
msg_level
0
1
0
0
3
4
0
3
1
<felds. . . >
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
msg_level
Critical
Error
Critical
Critical
3
4
Critical
3
Error
<felds. . . >
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
. . .
Exploring Splunk
52
I|gute 414. /J&> Command Ixamp|e
The /J&> command results create a new J/>"!)%8 ed. (l u J/>"!)%8
ed exsts, the /J&> command updates its value.) The /J&> command
creutes or overrdes ony one ed ut u tme.
rex
The =/_ commund extructs eds vhose vuue mutches u speced ler
Compatible Regular Expression (PCRE). (=/_ is shorthand for regular ex-
pression.)
What Are Regular Expressions?
1h|nl o| tegu|at exptess|ons as v||dcatds on s|eto|ds. \ouve ptobab|y |ooled
|ot ||es v||h exptess|ons ||le `.doc ot `.x|s. Regu|at exptess|ons |e| you |ale
|ha| |o a vho|e nev |eve| o| povet and |ex|b||||y. l| youte |am|||at v||h tegu|at
exptess|ons, youte ptobab|y no| tead|ng |h|s box. 1o |eatn mote, see http://www.
regular-expressions.info easily the best site on the topic.
1ab|e 41S. =/_ Command Ixamp|es
Command RcsuIt
c U =/_ [p="3- XqgC="3WEPY
G"- Xqg%"WEPY\
Extract C="3 and %" eds usng regu-
lar expressions. If a raw event con-
tains [p="3- H<$&# G"- r"B\, then
C="3QH<$&# and %"Qr"B.
=/_ C)/>*Q$&J/*$/&=!+N)*
Xqg<$/=W^(dYsXqg&AAW^(dYs
XqgH&J/*H/&=!+Z&3/W^(dY
Extract <$/=, &AA, and H&J/*H/&=!+I
Z&3/ rom u ed cued $&J/*I
$/&=!+N)*. If $&J/*$/&=!+N)* Q
[B"Bs$/&=!+s38N$&J/*N$/&=!+\,
then <$/=QB"B, &AAQ$/&=!+, and
H&J/*H/&=!+Z&3/Q38N$&J/*N
$/&=!+.
previous
search results
fnal results
eval velocity=distance/time
...|
distance
50
100
200
velocity
5
10
40
time
10
10
5
<felds. . . >
. . .
. . .
. . .
distance
50
100
200
time
10
10
5
<felds. . . >
. . .
. . .
. . .
53
=/_ 3"*/Q$/* [$FX^^*tMuIYt7uF
kkkkIkkkkIkkkkIFD\
Use $/* syntax to match the regex to
a series of numbers, and replace them
with an anonymized string.
lgure 4-l5 ustrutes the rst exumpe n 1ube 4-l5, extructng C="3 and
%" eds.
I|gute 41S. =/_ Command Ixamp|e
lookup
The >"".<A commund munuuy nvokes ed ookups rom u ookup
tube, enubng you to udd ed vuues rom un externu source. lor ex-
ample, if you have 5-digit zip codes, you might do a lookup on the street
name to apply a ZIP+4 9-digit zip code.
1ab|e 416. Command Ixamp|es
Command RcsuIt
c U >"".<A <$/=%"D="<A <$/=
&$ >"!&>N<$/= TKG@KG D="<A &$
<$/=ND="<A
lor u ookup tube vth eds <$/=
and D="<A, speced n stunzu nume
<$/=%"D="<A in %=&#$C"=3E!"#C,
1

look up the value of each events
>"!&>N<$/= ed. lor entres thut
match, the value of the lookup tables
group ed s vrtten to the event's
<$/=ND="<A ed.
previous
search results
rex "From: (?<from>.*)
To: (?<to>.*)
...|
fnal results
_raw
From: Susan To: Bob Subject: current to-do list Message Hi Bob, I wanted to
From: Meenu Subject: version 6 docs Message Hi Jan, Let's set up a time to
From: John To: Miguel Message Here's what we need to arrange for the . . .
_raw
Subject: current to-do list Message Hi Bob, I wanted to
Subject: version 6 docs Message Hi Jan, Let's set up a time to
Message Here's what we need to arrange for the . . .
from
Susan
Meenu
John
to
Bob
Miguel
1
Lookup tubes cun be congured through Munuger Lookups.
Exploring Splunk
54
c U >"".<A *#$>"".<A +"$%
TKG@KG )A
Cven u ed ookup numed *#$I
>"".<A, referencing a Python script
that performs a reverse DNS lookup
and accepts either a host name or
IP address as arguments, match the
host name values (+"$% ed n your
events to the host name values in the
table, and then add the correspond-
ing IP address values to your events
(in the )A ed).
c U >"".<A >"!&>Q%=</ <$/=I
`)A>"".<A <$/= &$ >"!&>N<$/=
TKG@KG `)A &$ <$/=N`)A
For a local lookup table that is pres-
ent only in the search head, look up
the value of each events <$/= ed.
For entries that match, the value of
the lookup tables `)A ed s vrtten
to the events <$/=N`)A ed.
lgure 4-l6 ustrutes the rst exumpe n 1ube 4-l6, >"".<A <$/=%"I
D="<A <$/= &$ >"!&>N<$/= TKG@KG D="<A &$ <$/=ND="<A.
I|gute 416. >"".<A Command Ixamp|e
This chapter has provided a crash course in the commands in the SPL.
The next chapter describes how you can enrich your data with tags and
event types and tell Splunk to watch for certain patterns and alert you
about them.
previous
search results
fnal results
lookup usertogroup user as local_user
OUTPUT group as user_group
...|
local_user
User1
User2
User3
user_group
C
E
F
<felds. . . >
. . ..
. . ..
. . ..
search results
user
User10
User9
User1
User7
User2
User3
User27
User98
group
A
B
C
D
E
F
G
H
<felds. . . >
. . ..
. . ..
. . ..
. . ..
. . ..
. . ..
. . ..
. . ..
usergroup lookup table
local_user
User1
User2
User3
<felds. . . >
. . ..
. . ..
. . ..
55
5 Enriching Your Data
To make your data more useable, add knowledge to it. What do we mean
by thut: \hen you te Spunk hov to extruct eds rom your dutu, you
cun sturt reusonng ubout those eds und gve Spunk the knovedge to
classify your data for deeper analysis. When you save reports and dash-
boards, your data becomes easier to understand for you and others. And
when you create alerts, Splunk proactively reveals potential issues so that
you dont have to look for them manually after the fact.
This chapter covers three areas:
- Using SpIunk to Undcrstand Data shows how to explore, catego-
rize, and become familiar with your data.
- DispIaying Data shows the basics of visualizing data.
- Crcating AIcrts about PotcntiaI ProbIcms shows how to track and
send alerts when metrics cross thresholds.
Using Splunk to Understand Data
\hen you rst encounter u nev source o muchne dutu, t cun ook ke u
mess of meaningless numbers and cryptic text. The more you know about
the system pumping out machine data, however, the more the data will
make sense to you. But even if you know a data set well, further explora-
tion can still bring new insights.
1he rst step n gettng to knov dutu s usng Spunk to denty eds n
the data. You can think of this like looking at all the pieces in a puzzle,
rst notcng ther shupes. 1he next step s to cutegorze dutu us u preum-
ble to aggregation and reporting. This is like sorting the puzzle pieces into
border pieces and interior pieces. The more you are able to understand
the data and piece the puzzle together, the clearer the picture becomes.
At last, the picture is complete (displaying the data) and you can share it
with others.
Exploring Splunk
56
Identifying Fields: Looking at the Pieces of the Puzzle
Splunk recognizes many common types of data, referred to as source
types. l you set the rght source type, Spunk cun use precongured set-
tngs to try to denty eds. 1hs s the cuse vth most types o veb server
logs, for example.
But there are often hidden attributes embedded in machine data. For ex-
ample, a product category may be part of a URL. By examining events
that have certain product categories in their URLs, you can determine
response times and error rates for different sections of the site or informa-
tion about which products are viewed the most.
Automatic Field Discovery
\hen you seurch, Spunk uutomutcuy extructs eds by dentyng
common patterns in the data, such as the presence of an equal sign (=)
between a key and a value. For example, if an event contains id=11
lname=smith Splunk automatically creates )* and >#&3/ eds thut
huve the exumpe vuues. And, us mentoned n Chupter 2, some eds
(such as $"<=!/, $"<=!/%8A/, +"$%, N%)3/, and >)#/!"<#%) are always
dented.
Oon| see vha| youte |ool|ng |ot: S|at| seatch|ng |ot ||. Sp|unl d|sp|ays on|y a
cet|a|n numbet o| |e|ds |n |he Il by de|au||. Hundteds mote may be ex|tac|ed
pet|ec||y. Seatch|ng |ot |hem bt|ngs |hem |o |he |op.
The ficId Discovcry switch on the ficIds sidebar in the UI turns this be-
huvor on und o. You cun see some seected eds (eds thut Spunk
seected by deuut or thut you huve seected), ooved by eds thut
Splunk pulled out because they appeared in multiple events. If you click
Edit, Spunk sts more eds thut you cun udd to the group o seected
eds. Cckng uny ed shovs you the top vuues extructed rom your
search results.
Iot mote |n|otma||on on au|oma||c |e|d ex|tac||on, see http://splunk.com/goto/
book#auto_felds.
Conguring Field Extraction
Congurng ed extructon cun huppen n tvo vuys. You cun et Spunk
uutomute the conguruton or you by usng the lnteructve led Lxtruc-
tor, or you cun munuuy specy the conguruton yourse.
Chapter 5: Enriching Your Data
57
The Interactive Field Extractor
From any event in your search results, you can start the lntcractivc ficId
fxtractor (IFX) by selecting fxtract ficIds from the fvcnt options menu,
which you reach by clicking the down arrow to the left of an event in the
events list (see Figure 5-1).
I|gute S1. Choos|ng Extract Fields |tom |he Event Options menu s|at|s |he
Interactive Field Extractor
The IFX appears in another tab or window in your browser. By entering
the kinds of values you seek (such as a client IP address in web logs),
Splunk generates a regular expression that extracts similar values (this is
especially helpful for the regular expression-challenged among us). You
cun test the extructon (to muke sure t nds the ed you're ookng or)
und suve t vth the nume o the ed.
1o |eatn mote abou| |he ln|etac||ve I|e|d Ix|tac|ot, see http://splunk.com/goto/
book#ifx.
Manually Conguring Field Extraction
From Managcr ficIds ficId cxtractions, you can manually specify reg-
uur expressons to extruct eds, vhch s u more exbe but udvunced
method or extructng eds.
1o |eatn abou| manua||y spec||y|ng tegu|at exptess|ons, see http://splunk.com/
goto/book#confg_felds.
Search Language Extraction
Another vuy to extruct eds s to use seurch communds. 1he most com-
mon command for extracting data is the =/_ command, described in the
ust chupter. lt tukes u reguur expresson und extructs eds thut mutch
that expression.
Exploring Splunk
58
Sometimes the command you use depends on the kind of data from
vhch you're extructng eds. 1o extruct eds rom mutne tubuur
events (such as command-line output), use 3<>%).J, and to extract from
XML and JSON data, use $A&%+ or _3>.J.
1o |eatn abou| commands |ha| ex|tac| |e|ds, see http://splunk.com/goto/
book#search_felds.
Exploring the Data to Understand its Scope
Ater eds ure extructed, you cun sturt exporng the dutu to see vhut t
tells you. Returning to our analogy of the puzzle, you begin by looking
or putterns. \hut peces hep dene the borders o the puzze: Hov ese
can you categorize the pieces? By shape or color?
The Scarch dashboards ficIds sidebar gives you some immediate infor-
muton ubout euch ed:
- 1he busc dutu type o the ed, ndcuted by u churucter to the et
o the ed nume (u s text und r s numerc).
- 1he number o occurrences o the ed n the events st (n puren-
theses oovng the ednume).
\hen you cck u ed nume n the ficIds sdebur, u summury o the ed
pops up, including top values and links to additional charts.
I|gute S2. V|ev a |e|d summaty by c||cl|ng on a |e|d name |n |he Fields sidebar.
59
You can also narrow the events list to see only events that have a value
or thut ed.
Exploring data using top
The %"A commund gves you the most common ed vuues, deuutng to
the top ten. You can use the top command to answer questions like these:
- What are my top 10 web pages?
$"<=!/%8A/Q\&!!/$$P\ U %"A <=)
- Who are the top users for each host?
$"<=!/%8A/Q\&!!/$$P\ U %"A <$/= B8 +"$%
- What are the top 50 source and destination IP pairs?
cU %"A >)3)%Q04 $=!N)A2 */$%N)A
Exploring data using stats
The $%&%$ command provides a wealth of statistical information about
your data.
Here are a few simple ways to use it:
- How many 503 response errors
2
have I had?
$"<=!/%8A/Q\&!!/$$P\ $%&%<$Q047 U $%&%$ !"<#%
- What is the average kilobytes per second for each host?
$"<=!/%8A/Q\&!!/$$P\ U $%&%$ &JDX.BA$Y B8 +"$%
- Hov muny peope bought overs yesterduy: Lse stuts dc (dstnct
count) to ensure that each IP address is counted only once.
$"<=!/%8A/Q\&!!/$$P\ &!%)"#QA<=!+&$/ !&%/D"=8N)*QC>"(/=$ U
$%&%$ *!X!>)/#%)AY
- What is the 95th percentile of time the servers took to respond to
web requests?
$"<=!/%8A/Q\&!!/$$P\ U $%&%$ A/=!:0X$A/#%Y
Adding sparklines to the mix
As of Splunk 4.3, you can add simple line graphs, known as sparklines,
to your tabular results. Sparklines let you quickly visualize a data pattern
without creating a separate line chart.
For example, this search uses sparklines to show the number of events
over time for each host:
P U $%&%$ $A&=.>)#/ !"<#% B8 +"$%
2
A status of 503 in web server logs is a server-side error. The web server
responded with a service unavailable message. The business meaning is that
someone came to your site and didnt get through. Its time to look at operations if
you keep seeing these errors.
Exploring Splunk
60
Figure 5-3 shows sparklines in the table.
I|gute S!. Spatl||nes shov pa||etns |n |he da|a |n |he Iven|s |ab|e
Here are a few more commands that demonstrate ways to use sparklines:
- What is the number of events for each status and category combina-
tion, over time?
$"<=!/%8A/Q\&!!/$$P\ U $%&%$ $A&=.>)#/ !"<#% B8 $%&%<$2
!&%/D"=8N)*
- What is the average time response time for each product category,
over time?
$"<=!/%8A/Q\&!!/$$P\ U $%&%$ $A&=.>)#/X&JDX$A/#%YY B8 !&%I
/D"=8N)*
Using a different data set (earthquake magnitude data), see how earth-
quake magnitude varies by region and over 6 hour chunks of time, with
the more popuur regons rst.
3

$"<=!/Q/v$1*&8IO6E0E!$J U $%&%$ $A&=.>)#/X&JDXO&D#)%<*/Y2;+Y
&$ 3&D#)%<*/N%=/#*2 !"<#%2 &JDXO&D#)%<*/Y B8 S/D)"# U $"=%
!"<#%
Preparing for Reporting and Aggregation
Ater you huve dented eds und expored the dutu, the next step s to
start understanding whats going on. By grouping your data into catego-
ries, you can search, report, and alert on those categories.
1he cutegores ve ure tukng ubout ure user-dened. You knov your dutu,
and you know what you want to get out of your data. Using Splunk, you
can categorize your data as many ways as you like.
There are two primary ways that Splunk helps with categorizing data: tag-
ging and event types.
3
We offer this as an example, but you can download real data and try it out by
going to: http://earthquake.usgs.gov/earthquakes/catalogs/.
61
Tagging
1ugs ure un eusy vuy to ube uny ed vuue. l the host nume B*DA"DI
)#I45 isnt intuitive, give it a tag, like &<%+/#%)!&%)"#N$/=J/=, to make it
more understandable. If you see an outlier value in the UI and want to be
able to revisit it later and get more context, you might label it C">>"(N<A.
1o tug u ed vuue n the events st, cck the dovn urrov besde the ed
value you want to tag (see Figure 5-4).
I|gute S4. 1agg|ng hos|s
You can manage all your tags by going to Managcr Tags.
Lets suppose youve labeled your various host values with tags such as
(/B$/=J/=, *&%&B&$/N$/=J/=, and so on. You can then report on those
custom tags to see your data the way you want instead of how it happens
to be named. Again, you decide how you want to look at your data. For
example, to compare how the various host types perform over time, run
a search such as:
c U %)3/!+&=% &JDX*/>&8Y B8 %&D--+"$%
Reporting and the Joy of Negative Searching
Itom |he momen| you s|at| |ool|ng a| da|a, you shou|d be |h|nl|ng abou|
reporting. What would you like to know about the data? What are you looking
|ot: \ha| no|se vou|d you ||le |o temove |tom |he da|a so |ha| you can eas||y
|nd vha| youte |ool|ng |ot:
1h|s |as| po|n| beats |ut|het exp|ana||on as an examp|e o| some|h|ng Sp|unl does
very well that few if any other data analysis software can: negative searching.
Click on the down arrow to tag the host
Exploring Splunk
62
Its often said that you cant prove a negative. You cant look everywhere and
say, vha| l seel |s no| |hete. \||h Sp|unl you can do nega||ve seatch|ng and
|n |ac| you shou|d. 1he teason ||s hatd |o see vha|s happen|ng v||h |og ||es,
and many o|het |ypes o| da|a, |s |ha| so much o| || |s |he same, sot| o| bus|ness
asusua| mach|ne da|a. \||h Sp|unl you can ca|egot|ze |ha| un|n|etes||ng da|a
and |e|| Sp|unl |o shov you on|y vha|s unusua| ot d|||eten|. Shov me vha| l
haven| seen be|ote. Some secut||y expet|s use Sp|unl |n jus| |h|s vay |o |den|||y
anoma|ous even|s |ha| cou|d |nd|ca|e an |n|tus|on, |ot examp|e. l| |heyve seen ||
be|ote, |hey g|ve || a |ag and exc|ude || |tom |he|t seatch. ||et you do |h|s |ot a
while, if anything odd happens, youll see it right away.
Event Types
When you search in Splunk, you start by retrieving events. You implicitly
look for a particular kind of event by searching for it. You could say that
you were looking for events of a certain type. Thats how event types
are used: they let you categorize events.
Event types facilitate event categorization using the full power of the
$/&=!+ command, meaning you can use Boolean expressions, wildcards,
ed vuues, phruses, und so on. ln ths vuy, event types ure even more
poveru thun tugs, vhch ure mted to ed vuues. ut, ke tugs, hov
your data is categorized is entirely up to you.
You might create event types to categorize events such as where a cus-
tomer purchased, when a system crashed, or what type of error condition
occurred.
Its all about what you need to know about your events.
Here ure some ground rues or u seurch thut denes un event type:
- No pipes. You cant have a pipe in a search used to create an event
type (i.e., it cannot have any search commands other than the im-
plied $/&=!+ command).
- No subseurches. At the end o Chupter 3, ve brey covered the
wheel-within-a-wheel that is subsearches; for now, remember that
you cant use them to create event types.
Heres a simple example. In our ongoing quest to improve our website,
were going to create four event types based on the $%&%<$ ed:
- stutus=2* s dened us success.
- stutus=3* s dened us redirect.
- stutus=4* s dened us client_error.
- stutus=5* s dened us server_error.
63
To create the event type $<!!/$$ us ve've dened t, you voud perorm
a search like this:
$"<=!/%8A/Q\&!!/$$P\ $%&%<$Q\6P\
Next, choose Crcatc fvcnt typc. The Savc As fvcnt Typc dialog appears
where you name the event type, optionally assign tags, and click Savc.
1o see |he even| |ypes ma|ch|ng yout seatch tesu||s, c||cl eventtype in the Fields
s|debat. 1h|s mu|||va|ued |e|d shovs a|| |he even| |ypes |ot |he even|s |n |he
events list.
We create the other three event types in just the same way, and then run
a $%&%$ !"<#% to see the distribution:
$"<=!/%8A/Q\&!!/$$P\U $%&%$ !"<#% B8 /J/#%%8A/
The results look like Figure 5-5.
I|gute SS. Bteal|ng dovn even|s by even| |ype
There are relatively few events with an event type of $/=J/=N/=="= but,
nonetheess, they mert u coser ook to see ve cun gure out vhut they
have in common.
Clicking $/=J/=N/=="= lets us to drill down into events of just that event
type, where we see 15 events that all look something like the one shown
in Figure 5-6.
Exploring Splunk
64
I|gute S6. n even| v||h a setvet ettot
The $/=J/=N/=="= events have one rather disturbing thing in common:
people are trying to buy something when the server unavailable status
occurs. In other words, this is costing us money! Its time to go talk to the
person vho udmnsters thut server und nd out vhut's vrong.
Nesting Event Types
\ou can bu||d mote spec||c even| |ypes on |op o| mote geneta| even| |ypes.
\e cou|d de|ne a nev even| |ype (/BN/=="= with other event types as building
blocks:
/J/#%%8A/Q!>)/#%N/=="= TS /J/#%%8A/Q$/=J/=N/=="=
Of course, you should use this sparingly because you dont want to risk losing
|tacl and |nadvet|en||y ctea||ng c|tcu|at de|n|||ons.
Tagging Event Types
Lvent types cun huve tugs (und so cun uny ed vuue or thut mutter). lor
example, you can tag all error event types with the tag /=="=. You can
then add a more descriptive tag about the types of errors relevant to that
event type. Perhaps there are three types of errors: one type that indicates
early warnings of possible problems, others that indicate an outage that
affects the user, and others that indicate catastrophic failure. You can add
another tag to the error event types that is more descriptive, such as /&=I
>8N(&=#)#D, <$/=N)3A&!%, or =/*N&>/=%, and report on them separately.
Together, event types and tags let you start building a higher-level model
from the detailed events of the machine data in question. Usually, this is
un terutve process. You begn by tuggng u ev useu eds, usng them
for monitoring and alerting. Soon after, youll create a few event types to
do more complex categorization. Perhaps you build higher-level event
types by referencing lower-level event types. Perhaps you then add tags
to your event types to unify several categorizations. All the while, youre
adding knowledge to Splunk about how to organize and label your data
for your needs.
65
Iat||et ve men||oned nega||ve seatch|ng. l| you |ag a|| |he even| |ypes you don|
especially want to see with a tag of #"=3&>, you can then search for events that
ate !C1 notma|. 1h|s bt|ngs abnotma||||es |o |he sut|ace.
ZTG %&D--/J/#%%8A/Q#"=3&>
Visualizing Data
So far weve shown you a couple of ways to get at data visualizations:
- Cckng u ednume n the ficIds sidebar to see some quick graph-
cs ubout u ed.
- Using the %"A and $%&%$ search commands.
- Using sparklines to see inline visualizations in the events table
results.
This section shows you how to create charts and dashboards for visual-
izing your data.
Creating Visualizations
When you look at a table of data, you may see something interesting.
Putting that same data into charts and graphs can reveal new levels of
information and bring out details that are hard to see otherwise.
To create charts of your data, after you run a search, select Crcatc
Rcport. Alternatively, in Splunk 4.3, click the RcsuIts Chart icon in the
Results area to display a chart of your results.
Splunk offers various chart types: column, line, area, bar, pie, and scat-
terplots.
What product categories are affected most by 404 errors? This search cal-
culates the number of events for each catcgory_id and generates the pie
chart shown in Figure 5-7.
$"<=!/%8A/Q\&!!/$$P\ $%&%<$Q\M4M\ U $%&%$ !"<#% B8 !&%/D"I
=8N)*
Exploring Splunk
66
I|gute S7. Iages no| |ound by ptoduc| ca|egoty
Cven thut overs und gts ure umong the hghest-murgn products, ve'd
better add some redirects for the bad URLs (and try to get the sites that are
linking to our pages to update their links).
\hen you mouse ovet any gtaph|c |n Sp|unl, you ge| mote |n|otma||on abou|
|he da|a beh|nd |ha| pot||on o| |he gtaph|c. See I|gute S8.
I|gute S8. Hovet|ng ovet pat| o| a gtaph|c d|sp|ays de|a|| abou| |he da|a
Hovering over part of a
graphic displays detail about
the data
67
Creating Dashboards
The end result of using Splunk for monitoring is usually a dashboard with
several visualizations. A dashboard is made up of report panels, which
can be a chart, a gauge, or a table or list of search results (often the data
itself is interesting to view).
When designing dashboards, ask yourself, Of all of these charts, which
ones voud l vunt to see rst: \hch ones voud end users vunt to
see rst: \hch ones voud ne-o-busness munugers vunt to see rst:
Maybe each audience needs its own dashboard.
Then you can ask, What questions arise from looking at this dashboard?
Splunk automatically handles many kinds of drill downs into chart spe-
ccs vth u smpe cck on the churt. (Advunced users cun specy dr-
down behavior explicitly, but that is beyond the scope of this book.)
One key point to remember is that simple visualizations are generally the
most popular with all levels of users. You can, and should, make more
advanced and detailed dashboards, but make sure to do a good job cov-
ering the simple, high-level views.
Figure 5-9 shows an example of a dashboard.
I|gute S0. dashboatd
The best way to build a dashboard is not from the top down but from the
bottom up, with each panel. Start by using Splunks charting capabilities
to show the vital signs in various ways. When you have several individual
charts showing different parts of the systems health, place them onto a
dashboard.
Exploring Splunk
68
Creating a Dashboard
In Splunk 4.3, to create a dashboard and add a report, chart, or search
results to it:
1. Run a search that generates a report for a dashboard.
2. Select Crcatc Dashboard pancI.
3. Give your search a name, and click Ncxt.
4. Decide if you want this report to go on a new dashboard or on an
existing dashboard. If youre creating a new dashboard, give it a
name. Click Ncxt.
5. Specify a title for your dashboard and a visualization (table, bar,
pie, gauge, etc.), and when you want the report for the panel to run
(vhenever the dushbourd s dspuyed or on u xed schedue).
6. Click Ncxt followed by the Vicw dashboard link or OK.
Viewing a Dashboard
At any time you can view a dashboard by selecting it from the Dash-
boards & Vicws menu at the top of the page.
Editing a Dashboard
While viewing your dashboard, you can edit it by clicking On in the Edit
mode selector and then clicking the Edit menu of any panel you want to
edit. From there, you can edit the search that generates a report or how
its visualized, or delete the panel.
Creating Alerts
What is an alert? You can think of an alert as an if-then statement that
gets evaluated on a schedule:
?C %+)$ +&AA/#$2 %+/# *" %+&% )# =/$A"#$/E
The if in this case is a search. The then is the action you want to be
tuken n response to the cuuse beng ued.
More formally, an alert is a search that runs periodically with a condition
evaluated on the search results. When the condition matches, some ac-
tions are executed.
Creating Alerts through a Wizard
1o get sturted vth creutng un uert, the rst step s to seurch or the con-
dition about which you want to be alerted. Splunk takes whatever search
69
is in the search bar when you create an alert and uses that as a saved
search, which becomes the basis for your alert (the if in your if-then).
With the search you want in the scarch bar, select Crcatc AIcrt. This
starts a wizard that makes it easy to create an alert.
Scheduling an Alert
On the SchcduIc screen of the Crcatc AIcrts diaIog, you name the alert
and specify how you want Splunk to execute it.
You can choose whether Splunk monitors for a condition by running a
search in real time, by running a scheduled search periodically, or by
monitoring in real time over a rolling window.
Here are the use cases for these three options:
- Monitor in real time if you want to be alerted whenever the condi-
tion happens.
- Monitor on a scheduled basis for less urgent conditions that you
nonetheless want to know about.
- Monitor using a real-time rolling window if you want to know if a
certain number of things happen within a certain time period (its a
hybrd o the rst tvo optons n thut sense). lor exumpe, trgger the
alert as soon as you see more than 20 404s in a 5-minute window.
If you specify that you want to monitor on a schedule or in a rolling win-
dow, you must also specify the time interval and the number of results
that should match the search to trigger the alert. Alternatively, you could
enter a custom condition, which is a search that is executed if the alert
condition is met. Custom conditions are described later in this chapter.
I|gute S16. Schedu||ng an a|et|
Exploring Splunk
70
The next step is to set limits and specify what to do if the alert is triggered.
Specifying Actions
What should happen if the alert condition occurs? On the Action screen
of the Crcatc AIcrt dialog, you specify what action or actions you want
to take (sending email, running a script, showing triggered alerts in Alerts
Manager).
In Figure 5-11, the user chose all of the above actions, letting us see all
the options available here.
I|gute S11. c||on scteen o| |he v|zatd
71
- Scnd cmaiI. Email has the following options:
^ Email addresses. Enter at least one.
^ Subject line. You can leave this as the default, which is Splunk
Alert: AlertName. The alert name is substituted for j#&3/j. (This
means you could change that subject to: Oh no! j#&3/j hap-
pened.)
^ Include the results that triggered the alert. Click the checkbox
to ncude them ether us un uttuched CSV e or seect inline to
put them right into the email itself.
- Run a script. You specify the script name, which must be placed in
Splunks home directory, within FB)#F$!=)A%$ or within an apps F
B)#F$!=)A%$ directory.
- Show triggcrcd aIcrts in AIcrt managcr, which you reach by click-
ing AIcrts in the upper right corner of the UI.
Ater you choose un ucton (or tvo or three), you cun n u ev more
options:
- Set the severity. The severity is metadata for your reference so that
you can classify alerts. The levels are info, low, medium, high, and
critical. Severity shows up in AIcrt managcr.
- Execute actions on all results or each result. This determines whether
Splunk takes the action (such as sending an email) for the group
of results that matches the search or for each individual result. All
results is the default.
- Throttling. Alerts are effective only if they tell you what you need to
know when you need to know it. Too many alerts and youll ignore
them. Too few and you wont know whats happening. This option
speces hov ong Spunk shoud vut to perorm the ucton ussoc-
ated with the alert again, after it has been triggered. If you specify a
rolling window, the wizard defaults the throttling interval to match
that window. More throttling options are described later in this
chapter.
After you click Ncxt, the nu step s to specy vhether the uert s prvute
or shared for read-only access to users of the current app. Click finish to
nuze the uert.
Tuning Alerts Using Manager
Setting the right limits for alerting usually requires trial and error. It may
take some adjustment to prevent too many unimportant alerts or too few
important ones. The limits should be tuned so that, for example, one spike
Exploring Splunk
72
in an isolated vital sign doesnt trigger an alert, but 10 vital signs getting
within 10% of their upper limits do.
Its easy to create alerts quickly using the wizard, but still more options for
tuning alerts are available using Manager.
Remember that saved searches underlie alerts. As a result, you edit them
like you would a saved search. To edit to your alert, choose Managcr and
then Scarchcs and Rcports.
Select a saved search from the list to display its parameters.
Setting Alert Conditions
1hnkng o un uert us un l-1hen stutement, you huve more exbty on
the If side by editing through the Manager. The alert can be set to trigger:
- Always
- Depending on the number of events, hosts, sources
- Custom condition
Although the wizard offered to alert on the number of events, here you
have options for alerting by the number of hosts or sources. Consider
hosts. Its one thing if you start seeing server unavailable status on one
web server in a cluster, but its quite another thing if you suddenly see it
on more and more of your servers. Clearly theres a spike and the servers
ure not hundng the truc.
1hs screen oers more exbty or denng the threshod or the uert:
- is greater than
- is less than
- is equal to
- not equal to
- rises by
- falls by
1he rst our optons vere exposed through the vzurd, but here ve udd
the ability to alert if the number rises or falls by a certain number or by
a certain percentage (such as 50%). rises by and falls by allow you
to effectively set alerts for conditions that are relative (its often not the
absolute number as much as a doubling or tripling that you want to be
alerted about). rises by and falls by are not supported on conditions
that use real-time searches.
73
Setting Custom Conditions
Athough the Ll oers exbty or congurng the most common knds
o uert condtons, sometmes you need even more exbty n the orm
of custom conditions.
A custom condition is a search against the results of the alerts main
seurch. l t returns uny resuts, the condton s true, und the uert s red.
For example, you might want to be alerted anytime a host goes down, but
exclude hosts that are undergoing scheduled maintenance. To do this,
youd make a main search to return all hosts that go down and a custom
condton ters out use postveshosts thut ure n the cuendur or
scheduled maintenance. In this way, you are alerted only if a host goes
down unexpectedly.
Throttling Alerts
Splunk lets you tune alerts so that they tell you something meaningful. A
message that tells you something important is helpful. One hundred mes-
suges, on the other hund, vhether usted or not, s not hepu. lt's nose.
Splunk lets you throttle alerts so that even if they are triggered, they go
o ony once n u purtcuur tme ntervu. ln other vords, the rst uert
s ke the rst kerne o popcorn thut pops, you don't vunt uerts or u
those other kernes, vhch ure reuy reuted to thut rst uert. (l popcorn
had a second alert, it should go off just after all functional kernels pop
and before any of them burn.)
This is what throttling does. You can tell Splunk to alert you but not to
keep alerting you.
In the middle of the Managers screen for editing alerts is an option called
AIcrt modc (see Figure 5-12).
I|gute S12. |et| Mode
You can be alerted once per search, that is, for all results, or you can be
uerted once per resut. ler resut uerts cun be urther throtted by eds.
For example, you may want to be alerted whenever the condition is ful-
ed, but ony once per host. Let's suy thut dsk spuce s runnng ov on u
Exploring Splunk
74
server and you want to be alerted when theres less than 30% free space
available. If you specify host in Pcr rcsuIt throttIing cIds, you would
ony be noted once or euch host durng the speced tme perod. l
you were dealing with user login failures, you might enter username as
the per-resut-throttng ed.
Customizing Actions for Alerting
By writing or modifying scripts, you can set up custom actions for alerts.
For example, you may want an alert to:
- Send an SMS to the people who can help with the problem.
- Create a helpdesk ticket or other type of trouble ticket.
- Restart the server.
All alert actions are based on a script, including sending an email. So is
creating an RSS feed. With that in mind, you can see that you can set up
uert uctons us exby us needed usng scrptng.
1o |eatn mote abou| ctea||ng cus|om a|et| sct|p|s, see http://splunk.com/goto/
book#custom_alerts
The Alerts Manager
Mission control for alerts is the AIcrt managcr.
Click Alert in the upper right corner of the screen to display the AIcrt
managcr.
75
I|gute S1!. |et| managet
A bre curcuton o termnoogy s needed here. \e' reer to the
suved -then schedued seurch us un uert, und un ndvduu rng o thut
alert as an alert instance.
The AIcrt managcr shovs the st o most recent rngs o uerts (.e.,
uert nstunces). lt shovs vhen the uert nstunce red, und provdes u
nk to vev the seurch resuts rom thut rng und to deete the rng. lt
also shows the alerts name, app, type (scheduled, real-time, or rolling
window), severity, and mode (digest or per-result). You can also edit the
uert's denton.
PART II
RECIPES
79
6 Recipes for Monitoring and
Alerting
1he rst ve chupters o ths book prepured you or usng Spunk to sove
problems, answer questions, and explore your data in new and interest-
ing ways.
In this chapter, well present monitoring and alerting recipes. Monitoring
refers to reports you can visually monitor and alerting refers to conditions
monitored by Splunk, which can automatically trigger actions.
These recipes are meant to be brief solutions to common monitoring and
alerting problems. Each recipe includes a problem statement followed
by a description of how to use Splunk to solve the problem. Some of
the more complex examples suggest variations on the recipe for you to
explore.
1o usk questons und nd more unsvers ke these, vst http://splunkbase.
com.
Monitoring Recipes
Monitoring can help you see what is happening in your data. How many
concurrent users are there? How are key metrics changing over time?
In addition to recipes that monitor various conditions, this section pro-
vdes recpes thut descrbe hov to use seurch communds to extruct eds
from semi-structured and structured data.
Monitoring Concurrent Users
Problem
You need to determine how many concurrent users you have at any par-
ticular time. This can help you gauge whether some hosts are overloaded
and enable you to better provision resources to meet peak demand.
Exploring Splunk
80
Solution
First, perform a search to retrieve relevant events. Next, use the !"#!<=I
=/#!8 commund to nd the number o users thut overup. lnuy, use
the %)3/!+&=% reporting command to display a chart of the number of
concurrent users over time.
Lets say you have the following events, which specify date, time, request
duration, and username:
0F54F54 5-44-45 S/vG)3/Q7 K$/=Qe$3)%+
0F54F54 5-44-45 S/vG)3/Q6 K$/=Q=%8>/=
0F54F54 5-44-45 S/vG)3/Q04 K$/=Q+e"#/$
0F54F54 5-44-55 S/vG)3/Q6 K$/=Q=()>>)&3$
0F54F54 5-44-56 S/vG)3/Q7 K$/=Q&A"#*
You can see that, at 1:00:01, there are three concurrent requests (e$3)%+,
=%8>/=, +e"#/$); at 1:00:11, there are two (+e"#/$, =()>>)&3$); and at
1:00:12, there are three (+e"#/$, =()>>)&3$, &A"#*).
Use this search to show the maximum concurrent users for any particular
time:
g8"<= $/&=!+ +/=/W $"<=!/%8A/Q>"D)#N*&%&
U !"#!<==/#!8 *<=&%)"#QS/vG)3/
U %)3/!+&=% 3&_X!"#!<==/#!8Y
1o |eatn mote abou| |he concuttency command, see http://splunk.com/goto/
book#concurrency
Monitoring Inactive Hosts
Problem
You need to determine which hosts have stopped sending data. A host
might stop logging events if the server, or application producing logs, has
crashed or been shut down. This often indicates a serious problem. If a
host stops logging events, youll want to know about it.
Solution
Use the 3/%&*&%& command, which reports high-level information about
hosts, sources, and source types in the Splunk indexes. This is what is
used to create the Summary Dashboard. Note the pipe character is at
the beginning of this search, because were not retrieving events from
a Splunk index, rather were calling a data-generating command (3/%&I
*&%&).
Chapter 6: Recipes for Monitoring and Alerting
81
Use the following search to take the information on hosts, sort it so the
eust recenty reerenced hosts ure rst, und dspuy the tme n u reudube
time format:
U 3/%&*&%& %8A/Q+"$%$
U $"=% =/!/#%G)3/
U !"#J/=% !%)3/X=/!/#%G)3/Y &$ w&%/$%NG)3/
Youll quickly see which hosts havent logged data lately.
1o |eatn mote abou| |he me|ada|a command, see http://splunk.com/goto/
book#metadata
Reporting on Categorized Data
Problem
You need to report on segments o your dutu thut uren't neuty dened.
Solution
1o seurch or specc purts o your dutu, cussy your events usng tugs
and event types. Tags are simpler but event types are more powerful (tags
and event types are discussed in Chapter 5).
\ou m|gh| vondet hov |h|s ca|egot|za||on o| da|a comes undet mon||ot|ng.
Thats because when you categorize data using tags and event types, you not
on|y ca|egot|ze |he da|a you have |oday, bu| you |each Sp|unl |o ca|egot|ze da|a
||le |ha| evety ||me || shovs up. \ou ate |each|ng Sp|unl |o be on |he |oolou| |ot
data that has certain characteristics. Think of tags and event types like putting out
an a|| po|n|s bu||e||n (IB) |ot yout da|a.
Using Tags
You cun cussy smpe ed=vuue purs usng tugs. lor exumpe, cussy
events that have +"$%Q*B4: as a *&%&B&$/ host by tuggng thut ed vuue.
This creates a %&D--+"$% ed huvng u vuue o *&%&B&$/, on events with
+"$%Q*B4:. You cun then use ths custom cusscuton to generute re-
ports.
Here are a couple of examples that use tags.
Show the top ten host types (good for bar or pie charts):
EEE U %"A 54 %&D--+"$%
Compare how the various host types perform over time:
Exploring Splunk
82
EEE U %)3/!+&=% &JDX*/>&8Y B8 %&D--+"$%
Using Event Types
When you use event types, instead of tags, to classify events, you are not
mted to u smpe ed=vuue. You cun use the u pover o the $/&=!+
command, including Boolean operations, phrase matching, and wild-
cards. You could make an event type called *&%&B&$/N+"$% vth u den-
tion of [+"$%Q*BP TS +"$%Q"=!>P\, and another event type called (/BN
+"$%. Repeat the same searches as you did for tags, but replace %&D--+"$%
with /J/#%%8A/. For example, to show the top ten event types:
EEE U %"A 54 /J/#%%8A/
ecuuse event types ure not specc to u dmenson, such us hosts, user
type, or error codes, they are all in a common namespace, jumbled to-
gether. A search for %"A /J/#%%8A/$ might return *&%&B&$/N+"$% and
(/BN/=="=, which is probably not what you want because youd be com-
purng uppes to orunges. lortunutey you cun ter vhch event types
you report on, using the /J&> command, if you use a common naming
convention for your event types.
As an example, using event types, compare how the various host types
perform (displayed as a timechart), using only event types that end in
N+"$%:
cU /J&> +"$%N%8A/$ Q 3JC)>%/=X3&%!+X/J/#%%8A/2 [N+"$%j\YY
U %)3/!+&=% &JDX*/>&8Y B8 +"$%N%8A/$
Comparing Todays Top Values to Last Months
Problem
You need to know the top N values today and how they compare to last
months values. This can answer questions like, which products, or data-
base errors, are suddenly becoming more popular than they used to be?
Solution
For this solution, well use the example of music data to show the top
10 most played artists today and their average position for the month.
Assume the events have an &=%)$% ed und u $&>/$ ed thut tes hov
many units were sold at a particular time. Well use the sum of $&>/$ as
our metric$<3X$&>/$Ybut we could use any other metric.
1he u seurch ooks duuntng ut rst, but you cun breuk t dovn nto
simple steps:
1. Get the monthly rankings by artist.
2. Get the daily rankings by artist and append them to the results.
83
3. Use stats to join the monthly and daily rankings by artist.
4. Use sort and eval to format the results.
Get the monthly rankings
Lse ths seurch to nd the l0 bggest monthy sues by urtst:
$"<=!/%8A/Q3<$)!N$&>/$ /&=>)/$%QI74*x*
U $%&%$ $<3X$&>/$Y &$ 3"#%+N$&>/$ B8 &=%)$%
U $"=% 54 I 3"#%+N$&>/$
U $%=/&3$%&%$ !"<#% &$ O"#%+S&#.
The /&=>)/$%QI74*x* tells Splunk to retrieve events starting at 30 days
ago (in other words, get events from the last month). $%&%$ calculates the
sums of sales for each artist as the 3"#%+N$&>/$ ed. You nov huve u rov
for each artist, with two columns: 3"#%+N$&>/$ and &=%)$%E $"=% 54
V 3"#%+N$&>/$ keeps only those rows with the ten largest 3"#%+N$&>/$
values, in sorted order from largest to smallest. The $%=/&3$%&%$ com-
mand adds one or more statistics to each event, based on the current
value of the aggregate at the time the event is seen (not on the results as
a whole, like the $%&%$ command does). Effectively, $%=/&3$%&%$ !"<#%
&$ O"#%+S&#. ussgns the rst resut O"#%+S&#.Q5, the second result O"#I
%+S&#.Q6, and so on.
Get yesterdays rankings
Make three small changes to the monthly-rankings search to get yester-
days rank:
- Change the value for /&=>)/$% from I74*x* to I5*x* to get the rank-
ings from yesterday.
- Change every instance of month in the search to day.
- Wrap the search in an &AA/#* command so that the results are ap-
pended to the resuts rom the rst seurch.
&AA/#* a
$/&=!+ $"<=!/%8A/Q3<$)!N$&>/$ /&=>)/$%QI5*x*
U $%&%$ $<3X$&>/$Y &$ *&8N$&>/$ B8 &=%)$%
U $"=% 54 I *&8N$&>/$
U $%=/&3$%&%$ !"<#% &$ L&8S&#.
b
Use stats to join the monthly and daily ranks by artist
Use the $%&%$ commund to on the resuts by urtst, puttng the rst
monthly and daily rankings into one result.
$%&%$ C)=$%XO"#%+S&#.Y &$ O"#%+S&#. C)=$%XL&8S&#.Y &$
L&8S&#. B8 &=%)$%
Exploring Splunk
84
Format the output
Finally, well calculate the difference in ranking between the monthly
und duy runk, sort the resuts by the duy runk, und dspuy the eds n
music billboard order (rank, artist, change in rank, old rank):
/J&> *)CCQO"#%+S&#.IL&8S&#.
U $"=% L&8S&#.
U %&B>/ L&8S&#.2 &=%)$%2 *)CC2 O"#%+S&#.
Summary
Putting it all together, the search is as follows:
$"<=!/%8A/Q3<$)!N$&>/$ /&=>)/$%QI74*x*
U $%&%$ $<3X$&>/$Y &$ 3"#%+N$&>/$ B8 &=%)$%
U $"=% 54 I 3"#%+N$&>/$ U $%=/&3$%&%$ !"<#% &$ O"#%+S&#.
U &AA/#* a
$/&=!+ $"<=!/%8A/Q3<$)!N$&>/$ /&=>)/$%QI5*x*
U $%&%$ $<3X$&>/$Y &$ *&8N$&>/$ B8 &=%)$%
U $"=% 54 I *&8N$&>/$ U $%=/&3$%&%$ !"<#% &$ L&8S&#.
b
U $%&%$ C)=$%XO"#%+S&#.Y &$ O"#%+S&#. C)=$%XL&8S&#.Y &$
L&8S&#. B8 &=%)$%
U /J&> *)CCQO"#%+S&#.IL&8S&#.
U $"=% L&8S&#.
U %&B>/ L&8S&#.2 &=%)$%2 *)CC2 O"#%+S&#.
Variations
Here, we used the sum of sales as our metric$<3X$&>/$Ybut we
could use any metric, such as 3)#X$&>/$Y, or change the time ranges to
compare last week to this week.
1o |eatn mote abou| |he s|teams|a|s command, see http://splunk.com/goto/
book#streamstats
Finding Metrics That Fell by 10% in an Hour
Problem
You want to know about metrics that have dropped by 10% in the last
hour. This could mean fewer customers, fewer web page views, fewer
data packets, and the like.
85
Solution
To see a drop over the past hour, well need to look at results for at least
the past two hours. Well look at two hours of events, calculate a sepa-
rate metric for each hour, and then determine how much the metric has
changed between those two hours. The metric were looking at is the
count of the number of events between two hours ago and the last hour.
This search compares the count by host of the previous hour with the cur-
rent hour und ters those vhere the count dropped by more thun l0%:
/&=>)/$%QI6+x+ >&%/$%Qx+
U $%&%$ !"<#% B8 *&%/N+"<=2+"$%
U $%&%$ C)=$%X!"<#%Y &$ A=/J)"<$2 >&$%X!"<#%Y &$ !<==/#% B8
+"$%
U (+/=/ !<==/#%FA=/J)"<$ g 4E:
1he rst condton (/&=>)/$%QI6+x+ >&%/$%Qx+) retrieves two hours worth
of data, snapping to hour boundaries (e.g., 2-4pm, not 2:01-4:01pm). We
then get a count of the number of those events per hour and host. Be-
cause there are only two hours (two hours ago and one hour ago), $%&%$
C)=$%X!"<#%Y returns the count from two hours ago and >&$%X!"<#%Y
returns the count from one hour ago. The (+/=/ clause returns only those
events where the current hours count is less than 90% of the previous
hours count (which shows that the percentage dropped 10%).
As an exercise for you, think about what will go wrong with this search
when the time span crosses midnight. Do you see how to correct it by
adding C)=$%XN%)3/Y to the rst $%&%$ command and sorting by that new
value?
Variations
Instead of the number of events, use a different metric, such as the av-
erage delay or minimum bytes per second, and consider different time
ranges, such as day over day.
Charting Week Over Week Results
Problem
You need to determine how this weeks results compare with last weeks.
Solution
First, run a search over all the events and mark whether they belong to
this week or last week. Next, adjust the time value of last weeks events
to look like this weeks events (so they graph over each other on the same
time range). Finally create a chart.
Exploring Splunk
86
Lets get results from the last two weeks, snapped to the beginning of the
week:
/&=>)/$%QI6(x( >&%/$%Qx(
Mark events as being from this week or last week:
/J&> 3&=./= Q )C XN%)3/ g =/>&%)J/N%)3/X#"(XY2 [I5(x(\Y2
[>&$% (//.\2 [%+)$ (//.\Y
Adjust last weeks events to look like they occurred this week:
/J&> N%)3/ Q )C X3&=./=QQ\>&$% (//.\2
N%)3/ d 1P6MP;4P;42 N%)3/Y
Chart the desired metric, using the week marker we set up, such as a
timechart of the average bytes downloaded for each week:
%)3/!+&=% &JDXB8%/$Y B8 3&=./=
This produces a timechart with two labeled series: last week and this
week.
Putting it all together:
/&=>)/$%QI6(x( >&%/$%Qx(
U /J&> 3&=./= Q )C XN%)3/ g =/>&%)J/N%)3/X#"(XY2 [I5(x(\Y2
[>&$% (//.\2 [%+)$ (//.\Y
U /J&> N%)3/ Q )C X3&=./=QQ\>&$% (//.\2
N%)3/ d 1P6MP;4P;42 N%)3/Y
U %)3/!+&=% &JDXB8%/$Y B8 3&=./=
If you use this pattern often, youll want to save it as a macro to reuse it.
Variations
Explore different time periods, such as day over day, with different chart
types. Try different charts other than &JDXB8%/$Y. Alternatively, remove
the snapping to week boundaries by setting /&=>)/$%QI6(, not using a
>&%/$% value (it defaults to now), and changing the =/>&%)J/N%)3/XY
argument to V5(.
Identify Spikes in Your Data
Problem
You want to identify spikes in your data. Spikes can show you where you
have peaks (or troughs) that indicate that some metric is rising or falling
shurpy. 1ruc spkes, sues spkes, spkes n the number o returns, spkes
in database loadwhatever type of spike you are interested in, you want
to watch for it and then perhaps take some action to address those spikes.
87
Solution
Use a moving trendline to help you see the spikes. Run a search followed
by the %=/#*>)#/ commund usng u ed you vunt to creute u trendne
for.
For example, on web access data, we could chart an average of the B8%/$
ed:
$"<=!/%8A/Q&!!/$$P U %)3/!+&=% &JDXB8%/$Y &$ &JDNB8%/$
To add another line/bar series to the chart for the simple moving average
(sma) of the last 5 values of B8%/$, use this command:
%=/#*>)#/ $3&0X&JDNB8%/$Y &$ 3"J)#DN&JDNB8%/$
If you want to clearly identify spikes, you might add an additional series
for spikeswhen the current value is more than twice the moving aver-
age:
/J&> $A)./Q)CX&JDNB8%/$ W 6 P 3"J)#DN&JDNB8%/$2 544442 4Y
The 10000 here is arbitrary and you should choose a value relevant to
your data that makes the spike noticeable. Changing the formatting of the
Y-axis to Log scale also helps.
Putting this together our search is:
$"<=!/%8A/Q&!!/$$P
U %)3/!+&=% &JDXB8%/$Y &$ &JDNB8%/$
U %=/#*>)#/ $3&0X&JDNB8%/$Y &$ 3"J)#DN&JDNB8%/$
U /J&> $A)./Q)CX&JDNB8%/$ W 6 P 3"J)#DN&JDNB8%/$2 544442 4Y
Variations
We used a simple moving average for the last 5 results ($3&0). Consider a
different number of values (for example, $3&64), and other moving aver-
age types, such as exponential moving average (/3&) and weighted mov-
ing average ((3&).
Alternatively, you can bypass the charting altogether and replace the
above /J&> with a (+/=/ cuuse to ter your resuts.
EEE U (+/=/ &JDNB8%/$ W 6 P 3"J)#DN&JDNB8%/$
And by looking at the table view or as an alert, youll only see the times
when the &JDNB8%/$ spiked.
1o |eatn mote abou| |he |tend||ne seatch command, see http://splunk.com/goto/
book#trendline
Exploring Splunk
88
Compacting Time-Based Charting
Problem
You would like to be able to visualize multiple trends in your data in a
small space. This is the idea behind sparklinessmall, time-based charts
displayed within cells of your results table. Sparklines were invented by
Edward Tufte and incorporated in Splunk 4.3.
Solution
To produce these sparklines in your tables, simply enclose your $%&%$ or
!+&=% functions in the $A&=.>)#/XY function.
Here, well use the example of web access logs. We want to create a
small graph showing how long it took for each of our web pages to re-
spond (ussumng the ed $A/#% is the amount of time spent serving that
veb puge). \e huve muny puges, so ve' sort them to nd the puges uc-
cessed the most (i.e., having the largest !"<#% values). The 03 tells Splunk
to show details down to a 5-minute granularity in the sparklines.
$"<=!/%8A/Q&!!/$$P
U $%&%$ $A&=.>)#/X&JDX$A/#%Y203Y2 !"<#% B8 C)>/
U $"=% I !"<#%
Run this search over the last hour. The result is a series of mini graphs
showing how long it took each page to load on average, over time.
Variations
Try using different functions other than &JD. Try using values different
than 03 for granularity. If you remove the 03 granularity altogether, Splunk
automatically picks the right value for the search timespan.
Reporting on Fields Inside XML or JSON
Problem
You need to report on data formatted in XML or JSON.
Solution
Use the $A&%+ command, introduced in Splunk 4.3, to extract values from
XML- and JSON-formatted data. In this example, well assume a source
type of book data in XML or JSON. Well run a search that returns XML
or JSON as the events text, and use the $A&%+ command to extract the
author name:
$"<=!/%8A/QB"".$
U $A&%+ "<%A<%Q&<%+"= A&%+Q!&%&>"DEB"".E&<%+"=
89
When called with no A&%+ argument, $A&%+ extructs u eds rom the
rst 5000 churucters, vhch s congurube, creutng eds or euch puth
element. Paths have the form C""EB&=EB&`. Each level can have an op-
tional array index, indicated by curly braces (e.g., C""t5uEB&=). All array
elements can be represented by empty curly brackets (e.g., C""tu). The
nu eve or XML queres cun uso ncude un uttrbute nume, uso en-
closed by curly brackets (e.g., C""EB&=tx%)%>/u) and prefaced with a x.
Ater you huve the extructed ed, you cun report on t:
EEE U %"A &<%+"=
Variations
An older search command called _3>.J extracts simple XML key-value
pairs. For example, calling EEE U _3>.J on events that have a value of
gC""WB&=gFC""W creutes u ed C"" with a value bar. Another older com-
mund thut extructs eds rom XML s _A&%+.
Extracting Fields from an Event
Problem
You want to search for a pattern and extract that information from your
events.
Solution
Lsng communds to extruct eds s convenent or qucky extructng
eds thut ure needed temporury or thut uppy to specc seurches und
are not as general as a source or source type.
Regular Expressions
The =/_ commund uctutes ed extructon usng reguur expressons.
For example, on email data, the following search extracts the from and to
eds rom emu dutu usng the =/_ command:
$"<=!/%8A/Q$/#*3&)>N$8$>"D
U =/_ [p="3- XqgC="3WEPY G"- Xqg%"WEPY\
Delimiters
l you're vorkng vth mutpe eds thut huve demters uround them,
use the /_%=&!% command to extract them.
Suppose your events look like this:
U+/)D+%-16U&D/-M7U#&3/-3&%% $3)%+U
Extract the /J/#% eds vthout demters usng:
EEE U /_%=&!% A&)=*/>)3Q\U\ .J*/>)3Q\-\
Exploring Splunk
90
The result is what you would expect:
+/)D+%Q162 &D/QM72 &#* #&3/Q3&%% $3)%+E
Variations
Try using 3<>%).J, $A&%+, or _3>.J.
Alerting Recipes
Recall from Chapter 5 that an alert is made up of two parts:
- A condition: An interesting thing you want to know about.
- An action: what to do when that interesting thing happens.
ln uddton, you cun use throttng to prevent over-rng o repeuted uerts
of the same type.
For example:
- I want to get an email whenever one of my servers has a load above
a certain percentage.
- I want to get an email of all servers whose load is above a certain
percentage, but dont spam my inbox, so throttle the alerts for every
24 hours.
Alerting by Email when a Server Hits a Predened
Load
Problem
You vunt to be noted by emu vhen u server oud goes ubove 80%.
Solution
The following search retrieves events with load averages above 80% and
calculates the maximum value for each host. The top source type comes
with the Splunk Unix app (available at splunkbase.com), and is fed data
from the Unix %"A command every 5 seconds:
$"<=!/%8A/Q%"A >"&*N&JDW94
U $%&%$ 3&_X>"&*N&JDY B8 +"$%
Set up the alert in the following way, using the instructions from Chapter
5:
- Alert condition: alert if the search returns at least one result.
- Alert actions: email and set subject to: Server load above 80%.
- Suppress: 1 hour.
91
Variations
Change alert conditions and suppression times
Alerting When Web Server Performance Slows
Problem
You vunt to be noted by emu vhenever the 95th percente response
time of your web servers is above a certain number of milliseconds.
Solution
The following search retrieves weblog events, calculates the 95th percen-
tile response time for each unique web address (<=)NA&%+), und nuy
ters out uny vuues vhere the 95th percente s ess thun 200 msec-
onds:
sourcetype=weblog
] stuts perc95(response_tme) AS resp_tme_95 by ur_puth
] vhere resp_tme_95>200
Set up the alert in the following way:
- Alert condition: alert if the search returns at least X results (the num-
ber o sov veb requests you thnk mert un uert beng red).
- Alert actions: email, with subject set to: Web servers running slow.
If youre running in the cloud (for example, on Amazon EC2),
maybe start new web server instances.
- Suppress: 1 hour.
Shutting Down Unneeded EC2 Instances
Problem
You want to shut down underutilized EC2 instances.
Solution
The following search retrieves weblog events and returns a table of hosts
that have fewer than 10000 requests (over the timeframe that the search
runs):
$"<=!/%8A/Q(/B>"D
U $%&%$ !"<#% B8 +"$%
U (+/=/ !"<#%g54444
Exploring Splunk
92
Set up the alert in the following way:
- Alert condition: alert if the search returns at least X results (the num-
ber o hosts you thnk mert un uert beng red).
- Alert actions: trigger a script that removes servers from the load bal-
ancer and shuts them down.
- Suppress: 10 minutes.
Converting Monitoring to Alerting
The monitoring recipes in this chapter produce useful reports, valuable
in themselves. But, if you take a second look, many of these can also be
the basis for setting up alerts, enabling Splunk to monitor the situation
for you.
Here ve' brey dscuss convertng u ev o the montorng recpes nto
alerts.
Monitoring Concurrent Users
This recipe can be made into an alert by using its search with a custom
alert condition of [(+/=/ 3&_X!"#!<==/#!8Y W 64\. This alerts you if too
many concurrent users are logged in.
Variations: Consider calculating the average concurrency as well and
alerting if the max is twice the average.
Monitoring Inactive Hosts
A custom alert condition of (+/=/ #"(XY I =/!/#%G)3/ W ;4P;4 alerts
you if a host has not been heard from in over an hour.
Comparing Todays Top Values to Last Months
A custom alert condition of (+/=/ *)CC g I54 alerts you if an artist shoots
to number 1 today and was not in the top 10 for the last month.
Variations: Use the same recipe to monitor HTTP status codes and report
u stutus code (e.g., 404) suddeny becomes sgncunty more, or ess,
prevalent than it was over the last month.
Find Metrics That Fell by 10% in an Hour
This recipe is already set up conveniently for an alert. Fire an alert when
any events are seen.
Variation: Fire only when more than N declines are seen in a row.
93
Show a Moving Trendline and Identify Spikes
The variation for this recipe is already set up conveniently for an alert.
Fire an alert when any events are seen.
Variations: Fire only when more than N spikes are seen in a time period
(e.g., 5 minutes).
You mght nd t u useu exercse to udd uertng to the remunng mon-
toring recipes.
95
7 Grouping Events
These recipes offer quick solutions to some of the most common, real-
world problems we see that can be solved by grouping events.
Introduction
There are several ways to group events. The most common approach uses
either the %=&#$&!%)"# or $%&%$ command. But when should you use
%=&#$&!%)"# and when should you use $%&%$?
The rule of thumb: If you can use $%&%$, use $%&%$. Its faster than %=&#$I
&!%)"#, especially in a distributed environment. With that speed, how-
ever, comes some limitations. You can only group events with $%&%$ if
they huve ut eust one common ed vuue und you requre no other
constraints. Typically, the raw event text is discarded.
Like $%&%$, the %=&#$&!%)"# command can group events based on com-
mon ed vuues, but t cun uso use more compex construnts such us
total time span of the transaction, delays between events within the trans-
action, and required beginning and ending events. Unlike $%&%$, %=&#$I
&!%)"# retuns the ruv event text und ed vuues rom the orgnu events,
but it does not compute any statistics over the grouped events, other than
the *<=&%)"# (the delta of the N%)3/ ed betveen odest und nevest
events in the transaction) and the /J/#%!"<#% (the total number of events
in the transaction).
The %=&#$&!%)"# commund s most useu n tvo specc cuses:
- \hen unque ed vuues (uso knovn us denters) ure not su-
cent to dscrmnute betveen dscrete trunsuctons. 1hs s the
cuse vhen un denter mght be reused, or exumpe n veb ses-
sons dented by cooke/cent ll. ln ths cuse, tmespuns or puuses
should be used to segment the data into transactions. In other cases,
vhen un denter s reused, or exumpe n DHCl ogs, u purtcuur
message may identify the beginning or end of a transaction.
- When it is desirable to see the raw text of the events rather than an
unuyss on the consttuent eds o the events.
Exploring Splunk
96
Again, when neither of these cases is applicable, it is a better practice
to use $%&%$, as search performance for $%&%$ is generally better than
%=&#$&!%)"#. Cten there s u unque denter, und $%&%$ can be used.
lor exumpe, to compute stutstcs on the duruton o trudes dented
by the unque denter %=&*/N)*, the following searches yield the same
answer:
c U %=&#$&!%)"# %=&*/N)*
U !+&=% !"<#% B8 *<=&%)"#
c U $%&%$ =&#D/XN%)3/Y &$ *<=&%)"# B8 %=&*/N)*
U !+&=% !"<#% B8 *<=&%)"#
1he second seurch s more ecent.
However, if %=&*/N)* values are reused but the last event of each trade is
indicated by the text END, the only viable solution is:
c U %=&#$&!%)"# %=&*/N)* /#*$()%+QRZL
U !+&=% !"<#% B8 *<=&%)"#
If, instead of an end condition, %=&*/N)* values are not reused within 10
minutes, the most viable solution is:
c U %=&#$&!%)"# %=&*/N)* 3&_A&<$/Q543
U !+&=% !"<#% B8 *<=&%)"#
Finally, a brief word about performance. No matter what search com-
mands you use, its imperative for performance that you make the base
seurch us specc us possbe. Consder ths seurch:
$"<=!/%8A/Q_ U %=&#$&!%)"# C)/>*Q)A 3&_A&<$/Q50$ U $/&=!+
)AQ5E6E7EM
Here we are retrieving all events of $"<=!/%8A/Q_, building up transac-
tions, and then throwing away any that dont have an )AQ5E6E7EM. If all
your events have the same )A value, this search should be:
$"<=!/%8A/Q_ )AQ5E6E7EM U %=&#$&!%)"# C)/>*Q)A 3&_A&<$/Q50$
This search retrieves only the events it needs to and is much more ef-
cent. More ubout ths s n lndng Specc 1runsuctons uter n ths
chapter.
Chapter 7: Grouping Events
97
Recipes
Unifying Field Names
Problem
You need to build transactions from multiple data sources that use differ-
ent ed numes or the sume denter.
Solution
1ypcuy, you cun on trunsuctons vth common eds ke:
c U %=&#$&!%)"# <$/=#&3/
But when the <$/=#&3/ denter s cued derent numes (ogn, nume,
user, owner, and so on) in different data sources, you need to normalize
the ed numes.
If sourcetype , only contains C)/>*N, and sourcetype r only contains
C)/>*Nr, creute u nev ed cued C)/>*Ny which is either C)/>*N, or
C)/>*Nr, depending on which is present in an event. You can then build
the transaction based on the value of C)/>*Ny.
$"<=!/%8A/Q, TS $"<=!/%8A/Qr
U /J&> C)/>*Ny Q !"&>/$!/XC)/>*N,2 C)/>*NrY
U %=&#$&!%)"# C)/>*Ny
Variations
Above we invoked !"&>/$!/ to use vhchever ed vus present on un
event, but sometimes you will need to use some logic to decide which
ed to use n unyng events. /J&>z$ )C or !&$/ functions may come in
handy.
Finding Incomplete Transactions
Problem
You need to report on incomplete transactions, such as users who have
logged in but not logged out.
Exploring Splunk
98
Solution
Suppose you are searching for user sessions starting with a login and end-
ing with a logout:
c U %=&#$&!%)"# <$/=)* $%&=%$()%+Q\>"D)#\
/#*$()%+Q\>"D"<%\
You would like to build a report that shows incomplete transactionsus-
ers who have logged in but not logged out. How can you achieve this?
The %=&#$&!%)"# commund creutes un nternu booeun ed numed
!>"$/*N%_# to indicate if a given transaction is complete or not. Normally
incomplete transactions are not returned, but you can ask for these evict-
ed partial transactions by specifying the parameter .//A/J)!%/*Q%=</.
Evicted transactions are sets of events that do not match all the trans-
action parameters. For example, the time requirements are not met in
un evcted trunsucton. 1runsuctons thut u u the requrements ure
murked us compete by huvng the ed !>"$/*N%_# set to 1 (rather than 4
or ncompete trunsuctons). So the puttern or ndng ncompete truns-
actions would generally be:
c U %=&#$&!%)"# g!"#*)%)"#$W .//A/J)!%/*Q%="$/*N%_#Q4
In our case, however, theres a wrinkle. An /#*$()%+ condition not match-
ing will not set the !>"$/*N%_#Q4 because events are processed from new-
est to oldest. Technically, the /#*$()%+ condition starts the transaction, in
terms o processng. 1o get uround ths, ve need to ter trunsuctons
based on the !>"$/*N%_# ed, us ve us muke sure thut our trunsuctons
dont have both a >"D)# and a >"D"<%:
c U %=&#$&!%)"# <$/=)* $%&=%$()%+Q\>"D)#\
/#*$()%+Q\>"D"<%\
.//A/J)!%/*Q%="$/*N%_#Q4 ZTG X>"D)# >"D"<%Y
Variations
A variation on this solution is to use $%&%$, if your transactions dont have
$%&=%$()%+F/#*$()%+ conditions or time constraints, and you dont care
about preserving the actual %=&#$&!%)"#. In this example, you just want
the <$/=)* of users who havent logged out.
lrst, ve cun seurch speccuy or ogn und ogout events:
&!%)"#Q\>"D)#\ TS &!%)"#Q\>"D"<%\
99
Next, for each <$/=)*, we use $%&%$ to keep track of the &!%)"# seen per
<$/=)*. ecuuse events ure n tme descendng order, the rst ucton s
the most recent.
c U $%&%$ C)=$%X&!%)"#Y &$ >&$%N&!%)"# B8 <$/=)*
Finally, we keep only events where the most recent user action was a
login:
c U $/&=!+ >&$%N&!%)"#Q\>"D)#\
At this point we have the list of all <$/=)* values where the last action
was a login.
Calculating Times within Transactions
Problem
You need to nd the duruton tmes betveen events n u trunsucton.
Solution
The basic approach is to use the /J&> command to mark the points in
time needed to measure the different durations, and then calculate the
durations between these points using /J&> after the %=&#$&!%)"# com-
mand.
Note: ln |h|s chap|et, samp|e even|s |n a |tansac||on ate numbeted so |ha| ve can
te|et |o |hem as even|1, even|2, and so on.
For example, suppose you have a transaction made up of four events,
uned by u common )* ed und you vunt to meusure the duruton o
phase1 and phase2:
a5b G</ h<> ; 4:-5;-44 )*Q567M $%&=% "C /J/#%E
a6b G</ h<> ; 4:-5;-54 )*Q567M A+&$/5- *" $"3/ ("=.E
a7b G</ h<> ; 4:-5;-M4 )*Q567M A+&$/6- *" $"3/ 3"=/E
aMb G</ h<> ; 4:-51-44 )*Q567M /#* "C /J/#%E
By default, the timestamp of this transaction-based event will be from
the rst event (eventl), und the duruton v be the derence n tme
between event4 and event1.
To get the duration of phase1, well need to mark timestamps for event2
and event3. /J&>z$ $/&=!+3&%!+ function works well for this example,
Exploring Splunk
100
but you have the full range of /J&> functions available to you for more
complex situations.
cU /J&> A5$%&=% Q )CX$/&=!+3&%!+X[A+&$/5\Y2 N%)3/2 #<>>XYY
U /J&> A6$%&=% Q )CX$/&=!+3&%!+X[A+&$/6\Y2 N%)3/2 #<>>XYY
Next we make the actual transactions:
c U %=&#$&!%)"# )* $%&=%$()%+Q\$%&=% "C /J/#%\
/#*$()%+Q[/#* "C /J/#%\
Finally we calculate the duration for each %=&#$&!%)"#, using the values
calculated above.
cU /J&> A5N*<=&%)"# Q A6$%&=% I A5$%&=%
U /J&> A6N*<=&%)"# Q XN%)3/ d *<=&%)"#Y I A6$%&=%
In this example, we calculated the time of the last event by added N%)3/
(the tme o the rst event) und uddng duruton to t. Cnce ve knev the
last events time, we calculated A6N*<=&%)"# as the difference between
the last event and the start of phase2.
Variations
By default, the %=&#$&!%)"# commund mukes mutvuued eds out o
the ed vuues seen n more thun one o u trunsucton's composte events,
but those values are just kept as an unordered, deduplicated bag of val-
ues. For example, if a transaction is made of 4 events, and those events
each have a #&3/ ed us oovsnume=mutt, nume=umy, nume=rory,
name=amythen the transaction made up of four events will have a
mutvuued ed #&3/ with values of amy, matt, and rory. Note that
weve lost the order in which the events occurred and weve missed an
amy! To keep the entire list of values, in order, use the 3J>)$% option.
Here, were building a transaction and keeping the list of times for its
events:
c U /J&> %)3/$QN%)3/ U %=&#$&!%)"# )* 3J>)$%Q\%)3/$\
From here we can add on /J&> commands to calculate differences. We
cun cucuute the tme betveen the rst und second event n the trunsuc-
tion as follows:
c U /J&> *)CCN5N6 Q 3J)#*/_X%)3/$25Y I 3J)#*/_X%)3/$24Y
Finding the Latest Events
Problem
You need to nd the utest event or euch unque ed vuue. lor exumpe,
when was the last time each user logged in?
101
Solution
At rst, you mght be tempted to use the %=&#$&!%)"# or $%&%$ command.
For example, this search returns, for each unique <$/=)*, the rst vuue
seen or euch ed:
c U $%&%$ C)=$%XPY B8 <$/=)*
Note thut ths seurch returns the rst vuue o euch ed seen or events
that have the same <$/=)*. It provides a union of all events that have that
user lD, vhch s not vhut ve vunt. \hut ve vunt s the rst event vth
a unique <$/=)*. The proper way to do that is with the */*<A command:
c U */*<A <$/=)*
Variations
If you want to get the oldest (not the newest) event with a unique <$/=)*,
use the $"=%B8 clause of the */*<A command:
c U */*<A <$/=)* $"=%B8 d N%)3/
Finding Repeated Events
Problem
You want to group all events with repeated occurrences of a value in or-
der to remove noise from reports and alerts.
Solution
Suppose you have events as follows:
6456I41I66 55-M0-67 !"*/Q67:
6456I41I66 55-M0-60 !"*/Q117
6456I41I66 55-M0-6; !"*/QI5
6456I41I66 55-M0-61 !"*/QI5
6456I41I66 55-M0-69 !"*/QI5
6456I41I66 55-M0-6: !"*/Q6:6
6456I41I66 55-M0-74 !"*/Q6:6
6456I41I66 55-M0-76 !"*/QI5
6456I41I66 55-M0-77 !"*/QMMM
6456I41I66 55-M0-70 !"*/QI5
6456I41I66 55-M0-7; !"*/QI5
Your goal is to get 7 events, one for each of the code values in a row: 239,
773, -1, 292, -1, 444, -1. You might be tempted to use the %=&#$&!%)"#
command as follows:
c U %=&#$&!%)"# !"*/
Exploring Splunk
102
Using %=&#$&!%)"# here is a case of applying the wrong tool for the job.
As long as we dont really care about the number of repeated runs of
duplicates, the more straightforward approach is to use */*<A, which
removes duplicates. By default, */*<A will remove all duplicate events
(vhere un event s u dupcute t hus the sume vuues or the speced
eds). ut thut's not vhut ve vunt, ve vunt to remove dupcutes thut
appear in a cluster. To do this, */*<A has a !"#$/!<%)J/Q%=</ option that
tells it to remove only duplicates that are consecutive.
c U */*<A !"*/ !"#$/!<%)J/Q%=</
Time Between Transactions
Problem
You want to determine the time between transactions, such as how long
its been between user visits to your website.
Solution
Suppose we have a basic %=&#$&!%)"# search that groups all events by
a given user (!>)/#%)AI!"".)/ pair), but splits the transactions when the
user is inactive for 10 minutes:
c U %=&#$&!%)"# !>)/#%)A2 !"".)/ 3&_A&<$/Q543
Ultimately, our goal is to calculate, for each !>)/#%)AI!"".)/ pair, the
difference in time between the end time of a transaction and the start
time of a more recent (i.e. previous in order of events returned) transac-
tion. That time difference is the gap between transactions. For example,
suppose we had two pseudo transactions, returned from most recent to
oldest:
G5- $%&=%Q54-74 /#*Q54-M4 !>)/#%)AQ& !"".)/Q_
G6- $%&=%Q54-54 /#*Q54-64 !>)/#%)AQ& !"".)/Q_
The gap in time between these two transactions is the difference between
the start time of T1 (10:30) and the end time of T2 (10:20), or 10 minutes.
The rest of this recipe explains how to calculate these values.
First, we need to calculate the end time of each transaction, keeping in
mnd thut the tmestump o u trunsucton s the tme thut the rst event
occurred and the duration is the number of seconds that elapsed between
the rst und ust event n the trunsucton:
c U /J&> /#*N%)3/ Q N%)3/ d *<=&%)"#
103
Next we need to add the start time from the previous (i.e., more recent)
transaction to each transaction. That will allow us to calculate the differ-
ence between the start time of the previous transaction and our calcu-
lated /#*N%)3/.
To do this we can use $%=/&3$%&%$ to calculate the last value of the
start time (N%)3/) seen in a sliding window of just one transaction
D>"B&>QC&>$/ &#* ()#*"(Q5and to ignore the current event in that slid-
ing window!<==/#%QC&>$/. In effect, were instructing $%=/&3$%&%$ to
look only at the previous events value. Finally, note that were specifying
that this window is only applicable to the given user (!>)/#%)AI!"".)/
pair):
c U $%=/&3$%&%$ C)=$%XN%)3/Y &$ A=/JN$%&=%%)3/
D>"B&>QC&>$/ ()#*"(Q5 !<==/#%QC&>$/
B8 !>)/#%)A2 !"".)/
At ths pont, the reevunt eds mght ook somethng ke ths:
G5- N%)3/Q54-44-4;2 *<=&%)"#QM2 /#*N%)3/Q54-44-54
G6- N%)3/Q54-44-452 *<=&%)"#Q62 /#*N%)3/Q54-44-47
A=/JN$%&=%%)3/Q54-44-4;
G7- N%)3/Q54-44-442 *<=&%)"#Q42 /#*N%)3/Q54-44-45
A=/JN$%&=%%)3/Q54-44-45
Nov, ve cun nuy cucuute the derence n tme betveen the pre-
vious transactions start time (A=/JN$%&=%%)3/) and the calculated /#*N
%)3/. That difference is the gap between transactions, the amount of time
(in seconds) passed between two consecutive transactions from the same
user (!>)/#%)AI!"".)/ pair).
c U /J&> D&AN%)3/ Q A=/JN$%&=%%)3/ V /#*N%)3/
Putting it all together, the search becomes:
c U %=&#$&!%)"# !>)/#%)A2 !"".)/ 3&_A&<$/Q543
U /J&> /#*N%)3/ Q N%)3/ d *<=&%)"#
U $%=/&3$%&%$ C)=$%XN%)3/Y &$ A=/JN$%&=%%)3/
D>"B&>QC&>$/ ()#*"(Q5 !<==/#%QC&>$/
B8 !>)/#%)A2 !"".)/
U /J&> D&AN%)3/ Q A=/JN$%&=%%)3/ I /#*N%)3/
At this point you can do report on D&AN%)3/ values. For example, what is
the biggest and average gap length per user?
c U $%&%$ 3&_XD&AN%)3/Y &$ 3&_2
&JDXD&AN%)3/Y &$ &JD
B8 !>)/#%)A2 !"".)/
Exploring Splunk
104
Variations
Given a simpler set of requirements, we can calculate the gaps be-
tween events in a much simpler way. If the only constraints for transac-
tions are $%&=%$()%+ and /#*$()%+meaning there are no time (e.g.,
3&_A&<$/Q543) or ed (e.g., !>)/#%)A, !"".)/) constraints then we can
calculate the gaps in transactions by simply swapping the $%&=%$()%+
and /#*$()%+ values.
For example, given these events:
54-44-45 >"D)#
54-44-46 >"D"<%
54-44-49 >"D)#
54-44-54 >"D"<%
54-44-50 >"D)#
54-44-5; >"D"<%
Rather than:
c U %=&#$&!%)"# $%&=%$()%+Q\>"D)#\ /#*$()%+Q\>"D"<%\
We can make the gaps between the standard transactions (login then
logout) be the transactions instead (logout then login):
c U %=&#$&!%)"# /#*$()%+Q\>"D)#\ $%&=%$()%+Q\>"D"<%\
From here the transactions are the gaps between logout and login events,
so we can subsequently calculate gap statistics using *<=&%)"#:
c U $%&%$ 3&_X*<=&%)"#Y &$ 3&_2 &JDX*<=&%)"#Y &$ &JD
Another vuruton on the theme o ndng tme betveen events s you
are interested in the time between a given event (event A) and the most
proximate newer event (event B). By using $%=/&3$%&%$, you can deter-
mine the range of times between the last two events, which is the differ-
ence between the current event and the previous event:
c U $%=/&3$%&%$ =&#D/XN%)3/Y &$ *<=&%)"# ()#*"(Q6
Finding Specic Transactions
Problem
You need to nd trunsuctons vth specc ed vuues.
Solution
A general search for all transactions might look like this:
$"<=!/%8A/Q/3&)>N>"D$ U %=&#$&!%)"# <$/=)*
105
Suppose, however, that we want to identify just those transactions where
there s un event thut hus the ed/vuue purs %"Q=""% and C="3Q3$3)%+.
You could use this search:
$"<=!/%8A/Q/3&)>N>"D$
U %=&#$&!%)"# <$/=)*
U $/&=!+ %"Q=""% C="3Q3$3)%+
The problem here is that you are retrieving all events from this sourcetype
(potentially billions), building up all the transactions, and then throwing
99% of the data right in to the bit bucket. Not only is it slow, but it is also
punuy necent.
You might be tempted to reduce the data coming in as follows:
$"<=!/%8A/Q/3&)>N>"D$ X%"Q=""% TS C="3Q3$3)%+Y
U %=&#$&!%)"# <$/=)*
U $/&=!+ %"Q=""% C="3Q3$3)%+
Athough you ure not necenty retrevng u the events rom the gven
sourcetype, there ure tvo uddtonu probems. 1he rst probem s utu:
you are getting only a fraction of the events needed to solve your prob-
em. Speccuy, you ure ony retrevng events thut huve u %" or a C="3
ed. Lsng ths syntux, you ure mssng u the other events thut coud
make up the transaction. For example, suppose this is what the full trans-
action should look like:
a5b 54F50F6456 54-55-56 <$/=)*Q567 %"Q=""%
a6b 54F50F6456 54-55-57 <$/=)*Q567 C="3Q3$3)%+
a7b 54F50F6456 54-55-5M <$/=)*Q567 $<Be/!%Q\$/=)"<$ /=="=\
aMb 54F50F6456 54-55-50 <$/=)*Q567 $/=J/=Q3&)>$/=J/=
a0b 54F50F6456 54-55-5; <$/=)*Q567 A=)"=)%8Q+)D+
The above search will not get event3, which has $<Be/!%, or event4,
which has $/=J/=, and it will not be possible for Splunk to return the
complete transaction.
The second problem with the search is that %"Q=""% might be very com-
mon and you could actually be retrieving too many events and building
too many transactions.
So what is the solution? There are two methods: using subsearches and
using the $/&=!+%_# command.
Exploring Splunk
106
Using Subsearches
Your goal is to get all the <$/=)* values for events that have %"Q=""%, or
C="3Q3$3)%+. Pick the more rare condition to get the candidate <$/=)*
values as quickly as possible. Lets assume that C="3Q3$3)%+ is more rare:
$"<=!/%8A/Q/3&)>N>"D$ C="3Q3$3)%+
U */*<A <$/=)*
U C)/>*$ <$/=)*
Now that you have the relevant <$/=)* values, you can search for just
those events thut contun these vuues und more ecenty bud the
transaction:
c U %=&#$&!%)"# <$/=)*
lnuy, ter the trunsuctons to muke sure thut they huve %"Q=""% and
C="3Q3$3)%+ (its possible that a <$/=)* value is used for other %" and
C="3 values):
c U $/&=!+ %"Q=""% ,ZL C="3Q3$3)%+
luttng ths u together, vth the rst seurch us u subseurch pussng the
userid to the outer search:
a
$/&=!+ $"<=!/%8A/Q/3&)>N>"D$ C="3Q3$3)%+
U */*<A <$/=)*
U C)/>*$ <$/=)*
b
U %=&#$&!%)"# <$/=)*
U $/&=!+ %"Q=""% C="3Q3$3)%+
Use searchtxn
The $/&=!+%_# (search transaction) command does the subsearch leg-
work for you. It searches for just the events needed to build a %=&#$&!I
%)"#. Speccuy, $/&=!+%_# does trunstve cosure o eds needed or
%=&#$&!%)"#, runnng the seurches needed to nd events or trunsucton,
then running the %=&#$&!%)"# seurch, und nuy terng them to the
speced construnts. l you vere unyng your events by more thun one
ed, the subseurch souton becomes trcky. $/&=!+%_# also determines
which seed condition is rarer to get the fastest results. Thus, your search
for email transactions with %"Q=""% and C="3Q3$3)%+, simply becomes:
U $/&=!+%_# /3&)>N%_# %"Q=""% C="3Q3$3)%+
107
But what is /3&)>N%_# in the above search? It refers to a transaction-type
denton thut hus to be creuted n u Spunk cong e%=&#$&!%)"#I
%8A/E!"#C. In this case, %=&#$&!%)"#%8A/E!"#C might look like:
a/3&)>N%_#b
C)/>*$Q<$/=)*
$/&=!+ Q $"<=!/%8A/Q/3&)>N>"D$
Running the $/&=!+%_# search will automatically run the search:
$"<=!/%8A/Q/3&)>N>"D$ C="3Q3$3)%+ U */*<A <$/=)*
The result of that search gives $/&=!+%_# the list of the <$/=)*$ to operate
upon. It then runs another search for:
$"<=!/%8A/Q/3&)>N>"D$ X<$/=)*Q567 TS <$/=)*Q7;: TS <$/I
=)*Q01; EEEY
U %=&#$&!%)"# #&3/Q/3&)>N%_#
U $/&=!+ %"Q=""% C="3Q3$3)%+
This search returns the needle-in-the-haystack transactions from the re-
sults returned by the $/&=!+%_# search.
Note: If the %=&#$&!%)"# commund's ed st hud more thun one ed,
$/&=!+%_# would automatically run multiple searches to get a transitive
closure of all values needed.
Variations
Lxpore usng mutpe eds vth the $/&=!+%_# command. If youre in-
terested in getting the relevant events and dont want $/&=!+%_# to actu-
ally build the transactions, use /J/#%$"#>8Q%=</.
Finding Events Near Other Events
Problem
You need to nd events beore und uter unother event. Suppose you vunt
to search for logins by root and then search backwards up to a minute for
unsuccessful root logins as well as forward up to a minute for changes in
passwords.
Solution
One solution is to use subsearches and look for the last instance of this
scenario. Do a subsearch for root logins and return $%&=%%)3/< and /#*I
%)3/<, which then scopes the parent search to those time boundaries
Exploring Splunk
108
when searching for either a C&)>/*N>"D)# or a A&$$("=*N!+&#D/* from
the same $=!N)A:
a
$/&=!+ $"<=!/%8A/Q>"D)#N*&%& &!%)"#Q>"D)# <$/=Q=""%
U /J&> $%&=%%)3/<QN%)3/ I ;4
U /J&> /#*%)3/<QN%)3/ d ;4
U =/%<=# $%&=%%)3/<2 /#*%)3/<2 $=!N)A
b
&!%)"#QC&)>/*N>"D)# TS &!%)"#QA&$$("=*N!+&#D/*
1he dovnsde to ths upprouch s thut t ony nds the ust nstunce o u
login and possibly has false positives, as it doesnt distinguish between
C&)>/*N>"D)#$ afterward or A&$$("=*N!+&#D/* before.
lnsteud, the probem cun be soved by terng the events dovn to ust
those we care about:
$"<=!/%8A/Q>"D)#N*&%& X &!%)"#Q>"D)# TS &!%)"#QC&)>/*N>"D)#
TS &!%)"#QA&$$("=*N!+&#D/* Y
The transaction should consist of events from the same $=!N)A that start
with a C&)>/*N>"D)# and end with a A&$$("=*N!+&#D/*. Furthermore, the
trunsucton shoud spun no more thun 2 mnutes rom sturt to nsh:
c U %=&#$&!%)"# $=!N)A 3&_$A&#Q63
$%&=%$()%+QX&!%)"#QC&)>/*N>"D)#Y
/#*$()%+QX&!%)"#QA&$$("=*N!+&#D/*Y
lnuy, you need to ter or ony those trunsuctons thut huve <$/=Q=""%.
Since a C&)>/*N>"D)# event often wont have <$/=Q=""% (the user hasnt
ogged n), t s necessury to ter uter the trunsucton:
c U $/&=!+ <$/=Q=""%
Conversely, if it was certain that <$/=Q=""% was in all the relevant events,
t shoud be udded to the seurch cuuse, skppng the nu terng ($/&=!+
<$/=Q=""%).
Finding Events After Events
Problem
You need to get the rst 3 events uter u purtcuur event (or exumpe, u
ogn event) but there s no ve-dened endng event.
109
Solution
Given the following ideal transaction that starts with a login action:
a5b 54-55-56 $=!N)AQ54E4E4E0 <$/=Q=""% &!%)"#Q>"D)#
a6b 54-55-57 $=!N)AQ54E4E4E0 <$/=Q=""% &!%)"#Q\!* F\
a7b 54-55-5M $=!N)AQ54E4E4E0 <$/=Q=""% &!%)"#Q\=3 I=C P\
aMb 54-55-50 $=!N)AQ54E4E4E0 <$/=Q=""% $/=J/=Q\/!+" >">\
The obvious search choice is to use %=&#$&!%)"# that $%&=%$()%+ the
login action:
EEE U %=&#$&!%)"# $=!N)A2 <$/= $%&=%$()%+Q\X&!%)"#Q>"D)#Y\
3&_/J/#%$QM
The problem is that you will get transactions that dont have &!%)"#Q>"D)#.
Why? The $%&=%$()%+ option does not tell %=&#$&!%)"# to return only
transactions that actually begin with the string youre supplying. Rath-
er it tells %=&#$&!%)"# that when it encounters a line that matches the
$%&=%$()%+ directive, it is the beginning of a new transaction. However,
transactions will also be made for different values of $=!N)A, regardless of
the $%&=%$()%+ condition.
1o uvod ths, udd u terng seurch commund uter the %=&#$&!%)"#
search above:
c U $/&=!+ &!%)"#Q>"D)#
The transactions returned will start with &!%)"#Q>"D)# and include the
next three events for the $=!N)A and <$/=.
Note: If there are less than three events between two logins, the transaction will
be sma||et |han 4 even|s. 1he %=&#$&!%)"# command adds an /J/#%!"<#% |e|d |o
each |tansac||on, vh|ch you can |hen use |o |ut|het |||et |tansac||ons.
Grouping Groups
Problem
You need to bud trunsuctons vth mutpe eds thut chunge vuue
within the transaction.
Exploring Splunk
110
Solution
Suppose you vunt to bud u trunsucton rom these our events, uned
by the +"$% and !"".)/ eds:
a5b +"$%Q&
a6b +"$%Q& !"".)/QB
a7b +"$%QB
aMb +"$%QB !"".)/QB
Because the value of +"$% changes during this transaction, a simple
%=&#$&!%)"# command unfortunately will make two distinct transactions:
c U %=&#$&!%)"# +"$%2 !"".)/
When it sees event1 and event2, it builds a transaction with +"$%Q&, but
when it gets to event3, which has a different value for host (+"$%QB), it
puts event3 and event4 into a separate transaction of events that have
+"$%QB. The result is that these four events are turned into two transac-
tions, rather than one transaction based on the common value of !"".)/:
Transaction1:
a5b +"$%Q&
a6b +"$%Q& !"".)/QB
Transaction2:
a7b +"$%QB
aMb +"$%QB !"".)/QB
You might be tempted to remove the +"$% ed rom the %=&#$&!%)"#
command and unify the transactions based on the !"".)/ value. The
problem is that this would create a transaction with event2 and event4,
ignoring event1 and event3 because they do not have a cookie value.
The solution to this problem is to build a transaction on top of a transac-
tion:
c U %=&#$&!%)"# +"$%2 !"".)/ U %=&#$&!%)"# !"".)/
This second %=&#$&!%)"# command will take the above two transactions
and unify them with a common !"".)/ ed.
Note thut you cure ubout the cucuuted eds *<=&%)"# and /J/#%I
!"<#%, they are now incorrect. The *<=&%)"# after the second %=&#$&!I
%)"# commund v be the derence betveen the trunsuctons t unes
rather than the events that comprise it. Similarly, the /J/#%!"<#% will be
the number o trunsuctons t uned, ruther thut the correct number o
events.
To get the correct /J/#%!"<#% uter the rst %=&#$&!%)"# command, cre-
ute u ed cued 38!"<#% to store all the /J/#%!"<#% values, and then
111
after the second %=&#$&!%)"# command sum all the 38!"<#% values to
calculate the =/&>N/J/#%!"<#%. Smury, uter the rst %=&#$&!%)"# com-
mand, record the start and end times of each transaction and then after
the second %=&#$&!%)"# command get the minimum start time and the
maximum end time to calculate the =/&>N*<=&%)"#:
c U %=&#$&!%)"# +"$%2 !"".)/
U /J&> 38!"<#%Q/J/#%!"<#%
U /J&> 38$%&=%QN%)3/
U /J&> 38/#*Q*<=&%)"# d N%)3/
U %=&#$&!%)"# !"".)/ 3J>)$%Q\38!"<#%\
U /J&> C)=$% Q 3)#X38$%&=%Y
U /J&> >&$%Q3&_X38/#*Y
U /J&> =/&>N*<=&%)"#Q>&$%IC)=$%
U /J&> =/&>N/J/#%!"<#% Q $<3X38!"<#%Y
113
8 Lookup Tables
1hese ookup tube recpes brey shov udvunced soutons to common,
reu-vord probems. Spunk's ookup euture ets you reerence eds
n un externu CSV e thut mutch eds n your event dutu. Lsng ths
mutch, you cun enrch your event dutu vth uddtonu eds. Note thut
we do not cover external scripted lookups or time-based lookups.
Introduction
These recipes extensively use three lookup search commands: >"".<A,
)#A<%>"".<A, and "<%A<%>"".<A.
lookup
lor euch event, ths commund nds mutchng rovs n un externu CSV
table and returns the other column values, enriching the events. For ex-
ample, an event with a +"$% ed vuue und u ookup tube thut hus +"$%
and 3&!+)#/N%8A/ ="($, specifying cU >"".<A 38>"".<A +"$% adds the
3&!+)#/N%8A/ value corresponding to the +"$% value to each event. By
default, matching is case-sensitive and does not support wildcards, but
you cun congure these optons. Lsng the >"".<A command matches
values in external tables explicitly. Automatic lookups, which are set up
using Splunk Manager, match values implicitly. To learn more about con-
gurng uutomutc ookups, see http://splunk.com/goto/book#autolookup.
inputlookup
This command returns the whole lookup table as search results. For ex-
ample, c U )#A<%>"".<A 38>"".<A returns a search result for each row in
the table 38>"".<A, vhch hus tvo ed vuues: +"$% and 3&!+)#/N%8A/.
outputlookup
You might wonder how to create a lookup table. This command outputs
the current search results to a lookup table on disk. For example, c U
"<%A<%>"".<A 38%&B>/E!$J $&J/$ all the results into 38%&B>/E!$J.
Exploring Splunk
114
Further Reading
http://splunk.com/goto/book#lookuptutorial
http://splunk.com/goto/book#externallookups
Recipes
Setting Default Lookup Values
Problem
You need u deuut ed vuue un event's vuue s not n the ookup tube.
Solution
There are several solutions.
Using an explicit >"".<A, you can simply use the /J&> !"&>/$!/ function:
c U >"".<A 38>"".<A )A U /J&> *"3&)#Q!"&>/$!/X*"3&)#2\<#.#"
(#\Y
Using automatic lookups, theres a setting for that. Go to Managcr >>
lookups >> lookup Dcnition >> myIookup, select the Advanccd op-
tions checkbox, and make the following changes:
Set Minimum matchcs: 1
Set DcfauIt matchcs: unknown
Save the changes.
Using Reverse Lookups
Problem
You need to search for events based on the output of a lookup table.
Solution
Splunk permits you to use reverse lookup searches, meaning you can
search for the output value of an automatic lookup and Splunk can trans-
ute thut nto u seurch or the correspondng nput eds o the ookup.
Chapter 8: Lookup Tables
115
For example, suppose you have a lookup table mapping 3&!+)#/N#&3/
to "(#/=:
3&!+)#/N#&3/2 "(#/=
(/B$/=J/=52/=).
*B$/=J/=12$%/A+/#
*B$/=J/=92&3=)%
c
If your events have a 3&!+)#/N#&3/ ed und you vunted to seurch or u
particular "(#/=, erik, you might use an expensive search, like this:
c U >"".<A 38>"".<A 3&!+)#/N#&3/ U $/&=!+ "(#/=Q/=).
This search is expensive because youre retrieving all of your events and
terng out uny thut don't huve erk us the ovner.
Aternutvey, you mght consder un ecent but compcuted subseurch:
c a )#A<%>"".<A 38>"".<A U $/&=!+ "(#/=Q/=). U C)/>*$ 3&I
!+)#/N#&3/b
1hs seurch retreves u the rovs o the ookup tube, ters out uny rovs
that dont have erik as the "(#/=, and returns a big OR expression of ma-
chine names for Splunk to ultimately run a search on.
But none of this is necessary. If youve set up an automatic lookup table,
you can simply ask Splunk to search for "(#/=Q/=)..
Thats it. Effectively, Splunk does the subsearch solution behind the
scenes, generating the search of OR clauses for you.
Note: Sp|unl a|so does au|oma||c tevetse seatch|ng |ot de|ned |e|d ex|tac||on,
tags, and eventtypesyou can seatch |ot |he va|ue |ha| vou|d be ex|tac|ed,
|agged, ot |yped, and Sp|unl te|t|eves |he cottec| even|s.
Variations
Using automatic lookups and the built-in reverse lookups, you can rec-
reate Splunks tagging system. For example, make a mapping from +"$%
to your ed cued +"$%N%&D. Now you can search for events based on
their +"$%N%&D and not only the +"$% vuue. Muny peope nd t euser to
maintain lookup tables than the Splunk tags.
Exploring Splunk
116
Using a Two-Tiered Lookup
Problem
You need to do a two-layered lookup. For example, look up an IP address
in a table of common, well-known hosts and, if that fails for a given event,
then and only then use a secondary, more expensive full DNS lookup.
Solution
Ater ve've retreved events, ve do our ntu ookup ugunst ocu_dns.
csv, u ocu ookup e:
EEE U >"".<A >"!&>N*#$ )A TKG@KG +"$%#&3/
If the lookup doesnt match, the +"$%#&3/ ed s nu or thut event.
We now perform the second, expensive lookup on events that have no
+"$%#&3/. By using TKG@KGZR{ instead of TKG@KG, the lookup will only run
on events that have a null value for +"$%#&3/.
EEE U >"".<A *#$>"".<A )A TKG@KGZR{ +"$%#&3/
Putting it all together:
EEE U >"".<A >"!&>N*#$ )A TKG@KG +"$%#&3/
U >"".<A *#$>"".<A )A TKG@KGZR{ +"$%#&3/
Using Multistep Lookups
Problem
You need to ook up u vuue n one ookup e und use u returned ed
vuue rom thut rst ookup to do u second ookup usng u derent ook-
up e.
Solution
You can do this manually by running sequential lookup commands. For
exumpe, u rst ookup tube tukes vuues o ed A und outputs vuues
o ed , und u second ookup tube tukes vuues o ed und outputs
vuues o ed C:
c U >"".<A 38NC)=$%N>"".<A , U >"".<A 38N$/!"#*N>"".<A r
More interestingly, this can be done using automatic lookups, where this
chaining happens automatically. It is imperative, however, that the look-
ups are run in the correct order, by using the alphanumeric precedence
of property names.
117
Go to Managcr >> lookups >> Automatic Iookups, and create two auto-
matic lookups, making sure that the one to run later has a named value
greater than the previous lookup name. For example:
4NC)=$%N>"".<A Q 38NC)=$%N>"".<A , TKG@KG r
5N$/!"#*N>"".<A Q 38N$/!"#*N>"".<A r TKG@KG m
Notc: Using lookup chaining as shown in this recipe, reverse lookups as in the
Is|ng Revetse Ioolups tec|pe do no| votl because Sp|unl |s cutten||y no|
ab|e |o tevetse mu|||p|e s|eps o| au|oma||c |e|d |oolups (e.g., au|oma||ca||y
convet||ng a seatch |ot cha|ned ou|pu| |e|d va|ue mQB&` into a search for input
|e|d va|ue ,QC"").
Creating a Lookup Table from Search Results
Problem
You want to create a lookup table from search results.
Solution
If you were to simply do:
g$"3/ $/&=!+W U "<%A<%>"".<A 38>"".<AC)>/E!$J
you mght encounter tvo probems. lrst, events huve muny eds, n-
cudng nternu eds ke N=&(, and N%)3/, which you dont want in your
ookup tube. Second, o the eds you do cure ubout, most key there
ure dupcute vuues on the events retreved. 1o hunde the rst probem,
we wont use the C)/>*$ command because its inconvenient to remove
nternu eds. lnsteud, ve' use the %&B>/ command to better limit the
eds to vhut ve expcty specy. 1o sove the second probem, use the
*/*<A command. Putting it all together:
c U %&B>/ C)/>*52 C)/>*6
U */*<A C)/>*5
U "<%A<%>"".<A 38>"".<AC)>/E!$J
Appending Results to Lookup Tables
Problem
You need to append results to an existing lookup table. For example, you
want to create a single lookup table based on the results of multiple itera-
tons o the sume seurch. Speccuy, suppose you vunted to keep truck
of the last IP each user logged in from. You might want to run a job every
15 minutes to look that up and update the lookup table with new users.
Exploring Splunk
118
Solution
The basic procedure is to get the set of results you want to append to
the lookup table, use )#A<%>"".<A to append the current contents of the
lookup, and use "<%A<%>"".<A to write the lookup. The command looks
like this:
8"<=N$/&=!+N%"N=/%=)/J/NJ&></$N#//*/*
U C)/>*$ %+/N)#%/=/$%)#DNC)/>*$
U )#A<%>"".<A 38>"".<A &AA/#*Q%=*$
U "<%A<%>"".<A 38>"".<A
lrst, ve tod Spunk to retreve the nev dutu und retun ony the eds
needed for the lookup table. Next, we used )#A<%>"".<A to append the
existing rows in 38>"".<A, by using the &AA/#*Q%=</ option. Next we re-
move duplicates with */*<A. Finally, we used "<%A<%>"".<A to output all
these results to 38>"".<A.
Variations
Suppose you want your lookup table to have only the most recent 30
days of values. You can set up a lookup table to be updated daily from a
scheduled search. When you set up your scheduled search to output the
lookup table and before the "<%A<%>"".<A command, add a condition
thut ters out dutu oder thun 30 duys:
EEE U (+/=/ N%)3/ WQ #"(XY I X;4P;4P6MP74Y
where 60*60*60*24*30 is the number of seconds in 30 days.
Building on the previous example, our search becomes:
8"<=N$/&=!+N%"N=/%=)/J/NJ&></$N#//*/*
U C)/>*$ e<$%N%+/N)#%/=/$%)#DNC)/>*$
U )#A<%>"".<A 38>"".<A &AA/#*Q%="".<A 38>"".<A
Obviously, youll also need to keep N%)3/ us one o the eds n your
lookup table.
Using Massive Lookup Tables
Problem
You have a massive lookup table but want performance to be fast.
119
Solution
When you have very large lookup tables and notice that performance is
affected, there are several solutions.
lrst, consder vhether you cun muke smuer, more specc ookup tu-
bles. For example, if some of your searches need only a subset of the rows
and columns, consider making a concise version of the lookup for those
searches. The following search reduced the size of 38>"".<A table by re-
ducing the rows to those that meet some condition, removing duplicates,
removng u coumns but u needed nput und output ed, und nuy
writing the results to the 38>"".<A6 table.
U )#A<%>"".<A 38>"".<A
U $/&=!+ $"3/!"#*)%)"#
U */*<A $"3/)#A<%C)/>*
U %&B>/ $"3/)#A<%C)/>*2 $"3/"<%A<%C)/>*
U "<%A<%>"".<A 38>"".<A6
If you cant reduce the size of the lookup table, there are other solutions.
If your Splunk installation has several indexers, those indexers automati-
cuy repcute your ookup tube. ut the ookup e s very urge (e.g.,
100MB), this may take a very long time long.
One solution, if your bundles are being frequently updated, is to disable
bundle replication and instead use NFS to make the bundles available to
all nodes.
See: http://splunk.com/goto/book#mount
Another solution, if your lookup table doesnt change too often and you
cannot rely on shared and mounted drives, is to use local lookups.
- To prevent the lookup from being replicated and distributed, add
the lookup table to the =/A>)!&%)"#r>&!.>)$% in *)$%$/&=!+E!"#C.
(See http://splunk.com/goto/book#distributed)
- Copy the ookup tube CSV e to euch o your ndexers n
jH@wKZoN|TORF/%!F$8$%/3F>"".<A
- When you run the search, add >"!&>Q%=</ option to the >"".<A
search command.
Note: Ioolup de|n|||ons de|ned |o |mp||c|||y tun v|a ptops.con| by |he|t vety
na|ute ate no| |oca| and mus| be d|s|t|bu|ed |o |ndexets.
lnuy, consder movng uvuy rom urge CSV es und consder usng
external lookups (usually leveraging a script that queries a database).
Exploring Splunk
120
Note: \hen a .csv |oolup |ab|e teaches a cet|a|n s|ze (16 MB by de|au||), Sp|unl
|ndexes || |ot |as|et access. By |ndex|ng |he .csv ||e, Sp|unl can seatch ta|het |han
scan |he |ab|e. 1o ed|| |he s|ze be|ote a ||e |s |ndexed, ed|| >)3)%$E!"#Czs lookup
stanza and change the 3&_N3/3%&B>/NB8%/$ value.
Comparing Results to Lookup Values
Problem
You want to compare the values in the lookup list with those in your
events. For example, you have a lookup table with IP addresses and want
to know which IP addresses occur in your data.
Solution
l events vth purtcuur ed vuues ure u smu subset o your events, you
cun ecenty use subseurches to nd reevunt events. Lse )#A<%>"".<A
in a subsearch to generate a large OR search of all the values seen in your
lookup table. The size of the list returned from a subsearch can be 10,000
tems n sze (modube n mts.con).
8"<=$/&=!+ a )#A<%>"".<A 38>"".<A U C)/>*$ )A b
The resulting search executed looks similar to:
8"<=$/&=!+ ,ZL X )AQ5E6E7EM TS )AQ5E6E7E0 TS EEE Y
You can test what the subsearch returns by running the search that is in-
side the subsearch and appending the C"=3&% command:
U )#A<%>"".<A 38>"".<A U C)/>*$ )A U C"=3&%
See: http://splunk.com/goto/book#subsearch
Variation I
Similarly, to retrieve events with values NOT in your lookup table, use a
pattern like:
8"<=$/&=!+ ZTG a )#A<%>"".<A 38>"".<A U C)/>*$ )A b
which results in a search running like this:
8"<=$/&=!+ ,ZL ZTG X )AQ5E6E7EM TS )AQ5E6E7E0 TS EEE Y
121
Variation II
Alternatively, if you want values in your lookup table that are not matched
in your data, use:
U )#A<%>"".<A 38>"".<A
U C)/>*$ )A
U $/&=!+ ZTG a $/&=!+ 8"<=$/&=!+ U */*<A )A U C)/>*$ )A b
vhch tukes u vuues n the ookup und ters out those thut mutch your
data.
Variation III
lor mussve sts, here s u trcky und ecent seurch puttern to nd u
the values in your events that are also in the lookup table: retrieve your
events and then append the entire lookup table to the events. By setting
u ed (e.g., 3&=./=), we can keep track of whether a result (think row)
is an event or a lookup table row. We can use $%&%$ to get the list of IP
uddresses thut ure n both sts (count>l):
8"<=$/&=!+
U /J&> 3&=./=Q*&%&
U &AA/#* a )#A<%>"".<A 38>"".<A U /J&> 3&=./=Q>"".<A b
U $%&%$ *!X3&=./=Y &$ >)$%N!"<#% B8 )A
U (+/=/ >)$%N!"<#% W 5
Note: ||hough |he append command appeats |o be execu||ng a subseatch, ||
|s no|. 1hete |s no ||m|| on |he numbet o| tesu||s appended, un||le a subseatch,
vh|ch has a de|au|| ||m|| o| 16l tesu||s.
If you need to use this technique over a very large timespan, it is more
ecent to use unother ookup tube to muntun ong-term stute. ln short,
schedule a search over a shorter time windowsuch as one daythat
calculates the last time an IP was seen. Then, use a combination of )#I
A<%>"".<A, */*<A, and "<%A<%>"".<A to incrementally update that lookup
table over the very long haul. This gives you a very quick resource to look
at to know the most recent state. See the Appending Results to Lookup
1ubes recpe or speccs.
Exploring Splunk
122
Controlling Lookup Matches
Problem
You have multiple entries in a lookup table for a given combination of
nput eds und vunt the rst vuue to mutch. lor exumpe, your ookup
tube mups hostnumes to severu host uuses, und you vunt the rst uus.
Solution
By default, Splunk returns up to 100 matches for lookups not involving a
time element. You can update it to return only one.
Using the UI, go to Managcr >> lookups >> lookup dcnitions and edit
or creute your ookup denton. Seect the Advanccd options checkbox
and enter 1 for Maximum matchcs.
Alternatively, you can edit the applicable %=&#$C"=3$E!"#C. Add 3&_N
3&%!+/$Q5 to your lookups stanza.
See: http://splunk.com/goto/book#feld_lookup
Variations
If your lookup table has duplicates that you want to remove, you can
clean them with a search similar to:
U )#A<%>"".<A 38>"".<A U */*<A +"$% U "<%A<%>"".<A 38>"".<A
1hs emnutes u but the rst dstnct occurrence o euch host n the e.
Matching IPs
Problem
You have a lookup table with ranges of IP addresses that you want to
match.
Solution
Suppose your events have IP addresses in them and you have a table of
IP ranges and ISPs:
#/%("=.N=&#D/2 )$A
664E5;0E:;E4F5:2 )$AN#&3/5
664E;ME5:6E4F5:2 )$AN#&3/6
EEE
You can specify a 3&%!+N%8A/ for a lookup. Unfortunately, this function-
ality isnt available in the UI but you can set it in the %=&#$C"=3$E!"#C
cong e.
123
Set the 3&%!+N%8A/ to m?LS for your #/%("=.N=&#D/.
In %=&#$C"=3$E!"#C:
a38>"".<Ab
3&%!+N%8A/ Q m?LSX#/%("=.N=&#D/Y
See: http://splunk.com/goto/book#transform
Variations
The available 3&%!+N%8A/ values are {?wLm,SL, m?LS, and Rk,mG. Rk,mG is
the deuut und does not need to be speced.
Also in %=&#$C"=3$E!"#C, you can specify whether lookup matching
should be case sensitive (the default) or not. To have matching be case
insensitive, use:
!&$/N$/#$)%)J/N3&%!+ Q p&>$/
Matching with Wildcards
Problem
You need wildcard matching for your lookup table.
Solution
Suppose you have a lookup table with URLs youd like to match on:
<=>2 &>>"(/*
PED""D>/E!"3FP2 G=</
(((EB>&!.>)$%E"=DP2 p&>$/
PF)3DFPeAD2 p&>$/
By including wildcard (P) characters in your lookup table values, you can
direct Splunk to match on wildcards.
As in the Matching IPs recipe, you can specify a 3&%!+N%8A/ for a look-
up in the %=&#$C"=3$E!"#C cong e:
a38>"".<Ab
3&%!+N%8A/ Q {?wLm,SLX<=>Y
Exploring Splunk
124
Note: By de|au|| |he max|mum ma|ches |ot |oolup |ab|es |s 166, so || you have
mu|||p|e tovs |ha| ma|ch, |he ou|pu| |e|ds v||| have mu|||p|e va|ues. Iot examp|e,
a ut| o| www.google.com/img/pix.jpg vou|d ma|ch |he |ts| and |h|td tov |n |he
|ab|e above, and |he a||oved |e|d vou|d become a mu|||va|ued |e|d v||h |he
va|ues 1tue and Ia|se. Isua||y |h|s |s no| vha| you van|. By se|||ng |he Maximum
matches se|||ng |o 1, |he |ts| ma|ch|ng va|ue v||| be used, and you case use |he
otdet o| |he |ab|e |o de|etm|ne ptecedence. \ou can |nd |h|s se|||ng a| Manager
>> Iookus >> Iooku Dcnlon >> mylooku, after selecting the Advanced
options checlbox.
Variations
1hs chupter's rst recpe deut vth deuut vuues vhen u ookup us
to match. Yet another way to accomplish this is with wildcard matching.
Make the last item in your lookup table have a match value of P, and set
the minimum and maximum matches for your lookup table to be 1.
125
Machinc-gcncratcd data has long been used in the data center by IT
professionals but has only recently been recognized as a new source
for helping other departments. Sometimes called IT data or operational
data, machine data is all of the data generated by applications, servers,
network devices, security devices, and other systems in your business.
1he unverse covered by muchne dutu s much more thun og est
ncudes dutu rom conguruton, cckstreums, chunge events, dugnos-
tics, APIs, message queues, and custom applications. This data is rigidly
structured, time-series based, and high-volume. Its generated by almost
every component in IT, and its formats and sources vary widely. Thou-
sands of distinct log formats, many from custom applications, are critical
to diagnosing service problems, detecting sophisticated security threats,
and demonstrating compliance. And with the explosion of connected de-
vices, the sheer amount of information being created by machines of all
kindsGPS devices, RFID tags, mobile phones, utility equipment, and
so onis expanding more quickly than our ability to process and use it.
The value of machine data is not news to IT professionals; they have used
t or yeurs. lncreusngy, users o Spunk nd thut t cun uso hep shed
ght on busness ssues. Muchne dutu s most oten stored n urge es,
and before Splunk, it would lie around dormant until problems arose
und these es hud to be munuuy nspected. \th Spunk these es ure
indexed and useable.
Business users are used to dealing with data generated by people par-
ticipating in business processes. Most often this transactional data, as its
called, is stored in one of two forms.
RcIationaI databascs are widely used to store transactional data. They
store structured enterprse dutu, such us nuncu records, empoyee re-
cords, manufacturing, logistical information, and the like. By design, re-
lational databases are structured with rigid schemas, or set of formulas
that describe the structure of a database. Changes to those schemas can
lead to broken functionality, introducing lengthy delays and risk when
making changes. To build a search in a relational database, practitioners
must usually make alterations to a schema.
Exploring Splunk
126
MuItidimcnsionaI databascs are designed for analyzing large groups of
records. The term OLAP (On-Line Analytical Processing) has become al-
most synonymous with multidimensional database. OLAP tools enable
users to analyze different dimensions of multidimensional data. Multidi-
mensional databases are great for data mining and monthly reporting, but
not for real-time events.
Machine data is at a much lower level of detail than transactional data.
Transactional data might store all of the product, shipping, and payment
data associated with an online purchase. The machine data associated
with this purchase would include thousands of records, or events, that
track every users click, every page and image loaded, every ad request-
ed, und so on. Muchne dutu s not ust ubout the nshed resut, or the
destination, but about the entire journey!
Because its so detailed, machine data can be used for a wide variety of
purposes. ln the vord o l1, muchne dutu cun, or exumpe, hep nd
problems and also show whether systems are operating within typical
ranges of performance. In the world of business, machine data can track
consumer behavior and help segment consumers for targeted marketing
messages.
To help you get a better idea of the nature of machine data, this appendix
brey descrbes some o the derent types you muy encounter.
Application Logs
Most homegrovn und puckuged uppcutons vrte ocu og es, oten
by logging services built into middlewareWebLogic, WebSphere
!
,
}oss, .NL1, lHl, und others. Log es ure crtcu or duy-to-duy de-
bugging of production applications by developers and application sup-
port. Theyre also often the best way to report on business and user activ-
ity and to detect fraud because they have all the details of transactions.
\hen deveopers put tmng normuton nto ther og events, og es
can also be used to monitor and report on application performance.
Web Access Logs
Web access logs report every request processed by a web serverwhat
client IP address it came from, what URL was requested, what the refer-
ring URL was, and data about the success or failure of the request. Theyre
most commonly processed to produce web analytics reports for market-
ingdaily counts of visitors, most requested pages, and the like.
Theyre also invaluable as a starting point to investigate a user-reported
problem because the log of a failed request can establish the exact time
127
of an error. Web logs are fairly standard and well structured. The main
challenge is in dealing with them is their sheer volume, as busy websites
typically experience billions of hits a day as the norm.
Web Proxy Logs
Nearly all enterprises, service providers, institutions, and government or-
ganizations that provide employees, customers or guests with web access
use some type of web proxy to control and monitor that access. Web
proxies log every web request made by users through the proxy. They
may include corporate usernames and URLs. These logs are critical for
monitoring and investigating terms of service abuses or corporate web
usage policy and are also a vital component of effective monitoring and
investigation of data leakage.
Call Detail Records
Call Detail Records (CDRs), Charging Data Records, and Event Data Re-
cords are some of the names given to events logged by telecoms and net-
work switches. CDRs contain useful details of a call or service that passed
through the switch, such as the number making the call, the number
receiving the call, call time, call duration, and type of call. As communi-
cations services move to Internet Protocol-based services, this data is also
referred to as IPDRs, containing details such as IP address, port number,
und the ke. 1he specs, ormuts, und structure o these es vury enor-
mously; keeping pace with all the permutations has traditionally been a
challenge. Yet the data they contain is critical for billing, revenue assur-
ance, customer assurance, partner settlements, marketing intelligence,
and more. Splunk can quickly index the data and combine it with other
business data to enable users to derive new insights from this rich usage
information.
Clickstream Data
Use of a web page on a website is captured in clickstream data. This pro-
vides insight into what a user is doing and is useful for usability analysis,
marketing, and general research. Formats for this data are nonstandard,
and actions can be logged in multiple places, such as the web server,
routers, proxy servers, and ad servers. Monitoring tools often look at a
purtu vev o the dutu rom u specc source. \eb unuytcs und dutu
warehouse products sample the data, thereby missing a complete view of
behavior and offering no real-time analysis.
Exploring Splunk
128
Message Queuing
Message queuing technologies such as TIBCO
, JMS, and AquaLogic

are used to pass data and tasks between service and application compo-
nents on a publish/subscribe basis. Subscribing to these message queues
is a good way to debug problems in complex applicationsyou can see
exactly what the next component down the chain received from the prior
component. Separately, message queues are increasingly being used as
the backbone of logging architectures for applications.
Packet Data
Data generated by networks is processed using tools such as tcpdump
und tcpov, vhch generute pcups dutu und other useu pucket-eve
and session-level information. This information is necessary to handle
performance degradation, timeouts, bottlenecks, or suspicious activity
that indicates that the network may be compromised or the object of a
remote attack.
Conguration Files
1here's no substtute or uctuu, uctve system conguruton to understund
hov the nrustructure hus been set up. lust congs ure needed or debug-
gng pust uures thut coud recur. \hen congs chunge, t's mportunt
to know what changed and when, whether the change was authorized,
and whether a successful attacker compromised the system to backdoors,
time bombs, or other latent threats.
Database Audit Logs and Tables
Databases contain some of the most sensitive corporate datacustomer
records, nuncu dutu, putent records, und more. Audt records o u du-
tabase queries are vital for understanding who accessed or changed what
data when. Database audit logs are also useful for understanding how
applications are using databases to optimize queries. Some databases log
uudt records to es, vhe others muntun uudt tubes uccessbe usng
SQL.
File System Audit Logs
1he senstve dutu thut's not n dutubuses s on e systems, oten beng
shared. In some industries such as healthcare, the biggest data leakage
rsk s consumer records on shured e systems. Derent operutng sys-
129
tems, third party tools, and storage technologies provide different options
or uudtng reud uccess to senstve dutu ut the e system eve. 1hs uudt
data is a vital data source for monitoring and investigating access to sensi-
tive information.
Management and Logging APIs
Increasingly vendors are exposing critical management data and log
events through standardized and proprietary APIs, rather than by them
oggng to es. Checkpont
revus og usng the ClSLC Log Lxport

API (OPSEC LEA). Virtualization vendors, including VMware
and Citrix
,
expose congurutons, ogs, und system stutus vth ther ovn Alls.
OS Metrics, Status, and Diagnostic Commands
Operating systems expose critical metrics, such as CPU and memory uti-
lization and status information using command-line utilities like A$ and
)"$%&% on Unix and Linux and perfmon on Windows. This data is usu-
ally harnessed by server monitoring tools but is rarely persisted, even
though its potentially invaluable for troubleshooting, analyzing trends to
discover latent issues, and investigating security incidents.
Other Machine Data Sources
There are countless other useful and important machine data sources we
did not describe, including source code repository logs, physical security
ogs, und so on. You st need revu und lDS ogs to report on netvork
connections and attacks. OS logs, including Unix and Linux syslog and
the Windows event logs, record who logged into your servers, what ad-
ministrative actions they took, when services start and stop, and when
kernel panics happen. Logs from DNS, DHCP, and other network ser-
vices record who was assigned what IP address and how domains are re-
solved. Syslogs from your routers, switches, and network devices record
the state of network connections and failures of critical network compo-
nents. Theres more to machine data than just logs and a much broader
diversity of logs than traditional Log Management solutions can support.
131
Appendix B: Case Sensitivity
Some things in Splunk are case-sensitive, while others are not, as sum-
marized in Table B-1.
1ab|e B1. Case sens|||v||y
Scnsitivc lnscnsitivc fxampIcs
Command names X
!"#$ &'($ )!*!)
Command key-
words
X
+, -)./ 01 )&*&)$ 2.3
4*5.$ 67
89 -)./ 01 )&*&)$
:;*2&$ &'($ 67
<=!> -)./ 01 2.(?*:.
Search terms X
.22'2$ @AA"A$ @22'2
Statistical func-
tions
X
*BC$ +DE$ +BC -)./ 01
)&*&)$ :;*2&$ 6
Boolean operators X
(uppercase)
+FG$ "A$ F"! H0''?.*4
'(.2*&'2)I
B)J *4/$ '2$ 4'& H?K&3
.2*? L.1M'2/)I
Field names X
;')& B)J >",!
Field values X
;')&N?':*?;')&$
;')&NO"P+O;')&
Regular expres-
sions
X
Q/Q/Q/ B)J QGQGQG
2.(?*:. com-
mand
X
.22'2 B)J @AA"A
133
Appendix C: Top Commands
Here are the most common search commands used by a sample of end-
users and by Splunk apps.
Top Scarchcs by fnd Uscrs Top Scarchcs by SpIunk Apps
Command Prevalence Command Prevalence
$/&=!+
10964
$/&=!+
1030
/J&>
4840
$%&%$
232
C)/>*$
2045
%)3/!+&=%
215
$%&%$
1840
/J&>
211
=/#&3/
1416
C)/>*$
142
%)3/!+&=%
1185
%"A
116
$"=%
1127
*/*<A
100
*/*<A
730
=/#&3/
96
C)>>#<>>
534
!+&=%
79
=/_
505
$"=%
76
%&B>/
487
=/_
42
!"#J/=%
467
+/&*
29
3/%&*&%&
451
3<>%).J
26
>"&*e"B
438
!">>/!%
25
!+&=%
437
$)%"A
21
(+/=/
384
!"#J/=%
20
&AA/#*
373
(+/=/
17
e")#
314
C)>>#<>>
17
+/&*
307
=/D/_
17
%"A
280
C"=3&%
16
%=&#$&!%)"#
260
>"".<A
14
Exploring Splunk
134
3&./3J
209
"<%>)/=
12
=&#D/3&A
202
e")#
9
&AA/#*!">$
201
=/A>&!/
9
>"".<A
157
$%=/&3$%&%$
8
=/A>&!/
102
135
Appendix D: Top Resources
We realize that this book cant tell you everything you need to know
about Splunk. Here is a list of websites to continue your education. These
links are also listed at http://splunk.com/goto/book#links.
Splunk download page
http://splunk.com/download
Splunk docs
http://docs.splunk.com
Splunk community
http://splunkbase.com
Community-based docs
http://innovato.com
Training videos
http://splunk.com/view/SP-CAAAGB6
Splunk videos
http://splunk.com/videos
Splunk blogs
http://blogs.splunk.com
Splunk TV
http://splunk.tv
137
Appendix E: Splunk Quick
Reference Guide
CONCEPTS
Overview
lndcx-timc Proccssing: Splunk reads data from a source, such as a
e or port, on u host (e.g. my muchne), cusses thut source into a
$"<=!/%8A/ (such as $8$>"D, &!!/$$N!"3B)#/*, or &A&!+/ /=="=), then
extracts timestamps, breaks up the source into individual events (such as
log events, alerts) which can consist of single or multiple lines, and writes
each event into an |ndex on disk, for later retrieval with a search.
Scarch-timc Proccssing: When a search starts, matching indexed events
are retrieved from disk, |e|ds (such as !"*/QM4M or <$/=Q*&J)*2EEE) are
extracted from the events text, and the event s cussed by mutchng
against /J/#%%8A/ dentons (such us /=="= or >"D)#). The events re-
turned from a search can then be powerfully transformed using SPL to
generate reports that display on dashboards.
Events
An event is one line of data. Here is an event in a web activity log:
517E6;E7ME667 I I a45Fh<>F644:-56-40-61 I4144b [iRG F%=&*/F
&AAq&!%)"#Q>"D"<% |GG@F5E5\ 644 6:07
More speccuy, un event s u set o vuues ussocuted vth u tmestump.
While many events are short and only take up a line or two, others can
be ong, such us u vhoe text document, u cong e, or vhoe }uvu stuck
trace. Splunk uses line-breaking rules to determine how it breaks these
events up for display in the search results.
Exploring Splunk
138
Sources and Sourcetypes
A source s the nume o the e, streum, or other nput rom vhch un
event originatesfor example, FJ&=F>"DF3/$$&D/$ or KL@-05M. Soutces
ure cussed nto sourcetypes, which may be well-known, such as &!I
!/$$N!"3B)#/* (veb server ogs) or cun be creuted on the y by Spunk
when it sees a source with data and formatting it hasnt seen before. Iven|s
with the same sourcetype can come from different sourcesevents from
the e FJ&=F>"DF3/$$&D/$ and from a syslog input on <*A-05M can both
have $"<=!/%8A/Q>)#<_N$8$>"D.
Hosts
A host is the name of the physical or virtual device from which an event
orgnutes. Hosts provde un eusy vuy to nd u dutu orgnutng rom u
particular device.
Indexes
When you add data to Splunk, Splunk processes it, breaking the data into
individual events, timestamps the events, and stores them in an |ndex so
that the data can be searched and analyzed later. By default, data you
feed to Splunk is stored in the 3&)# |ndex, but you can create and specify
other indexes for Splunk to use for different data inputs.
Fields
I|e|ds are searchable name/value pairings in event data. As Splunk pro-
cesses events at index time and search time, it automatically extracts
eds. At ndex tme, Spunk extructs u smu set o deuut eds or euch
event, including +"$%, $"<=!/, and $"<=!/%8A/. At search time, Splunk
extructs vhut cun be u vde runge o eds rom the event dutu, ncud-
ng user-dened putterns und obvous ed nume/vuue purs such us
<$/=)*Qe*"/.
Tags
Tags are aliases to |e|d values. For example, if two host names refer to
the same computer, you could give both host values the same tag (for
example, +&>:444). When you search for %&DQ+&>:444, Splunk returns
events involving both host name values.
139
Event Types
Iven| |ypes are dynamic tags attached to an event, if it matches the search
denton o the event type. lor exumpe, you dene un event type
called A="B>/3 vth u seurch denton o /=="= TS (&=# TS C&%&> TS
C&)>, whenever a search result contains /=="=, (&=#, C&%&>, or C&)>, the
event has an /J/#%%8A/ ed/vuue vth /J/#%%8A/QA="B>/3. If you were
searching for >"D)#, the logins with problems would be annotated with
/J/#%%8A/QA="B>/3. Iven| |ypes are cross-referenced searches that cat-
egorize events at search time.
Reports and Dashboards
Search results with formatting information (e.g., as a table or chart) are
informally referred to as reports, and multiple reports can be placed on a
common page, called a dashboard.
Apps
Apps ure coectons o Spunk congurutons, obects, und code. Apps
allow you to build different environments that sit on top of Splunk. You
can have one app for troubleshooting email servers, one app for web
analysis, and so on.
Permissions/Users/Roles
Saved Splunk objects, such as $&J/*$/&=!+/$, /J/#%%8A/$, =/A"=%$, and
%&D$, enrich your data, making it easier to search and understand. These
objects have petm|ss|ons and can be kept private or shared with other
users by roles (such as admin, power, or user). A role is a set of capabili-
tes thut you dene, such us vhether u purtcuur roe s uoved to udd
data or edit a report. Splunk with a free license does not support user
authentication.
Transactions
A transaction is a set of events grouped into one for easier analysis. For
example, because a customer shopping online generates multiple web
access events with the same SessionID, it may be convenient to group
those events into one transaction. With one transaction event, its easier
to generate statistics such as how long shoppers shopped, how many
items they bought, which shoppers bought items and then returned them,
and so on.
Exploring Splunk
140
Forwarder/Indexer
A forwarder is a version of Splunk that allows you to send data to a cen-
tral Splunk |ndexet or group of |ndexets. An |ndexet provides indexing
capability for local and remote data.
SPL
A search is a series of commands and arguments, chained together with
pipe character (U) that takes the output of one command and feeds it into
the next command.
$/&=!+I&=D$ U !3*5 !3*I&=D$ U !3*6 !3*I&=D$ U EEE
Seatch commands ure used to tuke ndexed dutu und ter unvunted
information, extract more information, calculate values, transform them,
and statistically analyze results. The search results retrieved from the in-
dex can be thought of as a dynamically created table. Each search com-
mund redenes the shupe o thut tube. Luch ndexed event s u rov, vth
141
coumns or euch ed vuue. Coumns ncude busc normuton ubout
the data and data dynamically extracted at search-time.
At the head of each search is an implied search-the-index-for-events com-
mand, which can be used to search for keywords (e.g., /=="=), boolean
expressions (e.g., X/=="= TS C&)><=/Y ZTG $<!!/$$Y, phrases (e.g., [*&I
%&B&$/ /=="=\), wildcards (e.g., C&)>P matches fail, fails, and failure),
ed vuues (e.g., !"*/QM4M), inequality (e.g., !"*/}QM4M or !"*/W644), a
ed huvng uny vuue or no vuue (e.g., !"*/QP or ZTG !"*/QP). For ex-
ample, the search:
$"<=!/%8A/Q\&!!/$$N!"3B)#/*\ /=="= U %"A 54 <=)
retrieves indexed &!!/$$N!"3B)#/* events from disk that contain the
term /=="= (ANDs are implied between search terms), and then for those
events, reports the top 10 most common URI values.
Subsearches
A subsearch is an argument to a command that runs its own search, re-
turning those results to the parent command as the argument value. Sub-
searches are enclosed in square brackets. For example, this command
nds u sysog events rom the user vth the ust ogn error:
$"<=!/%8A/Q$8$>"D a$/&=!+ >"D)# /=="= U =/%<=# <$/=b
Note that the subsearch returns one user value because by default the
=/%<=# command returns one value, although there are options to return
more (e.g., U =/%<=# 0 <$/=).
Relative Time Modiers
Besides using the custom-time ranges in the user interface, you can spec-
ify in your search the time ranges of retrieved events with the >&%/$% and
/&=>)/$% seurch moders. 1he reutve tmes ure speced vth u strng o
characters that indicate amount of time (integer and unit) and, optionally,
a snap to time unit:
adUIbg%)3/N)#%/D/=Wg%)3/N<#)%Wxg$#&AN%)3/N<#)%W
For example, /=="= /&=>)/$%QI5*x* >&%/$%QI5+x+ retrieves events con-
taining /=="= that from yesterday (snapped to midnight) to the last hour
(snapped to the hour).
Timc Units: Speced us second ($), minute (3), hour (+), day (*), week
((), month (3"#), quarter(v), or year (8). The preceding value defaults to 1
(i.e., 3 is the same as 53).
Exploring Splunk
142
Snapping: Indicates the nearest or latest time to which your time amount
rounds down. Snapping rounds down to the most recent time that is not
uter the speced tme. lor exumpe, t's ll:59:00 und you snup to
hours Xx+), you snap to 11:00, not 12:00. You can snap to a day of the
week, too; use x(4 for Sunday, x(5 for Monday, and so on.
COMMON SEARCH COMMANDS
COMMAND
!+&=%F%)3/!+&=%
Returns results in a tabular output for (time series)
charting.
*/*<A
Removes subsequent results that match.
/J&>
Calculates an expression. (See EVAL FUNCTIONS
table.)
C)/>*$
Removes eds rom seurch resuts.
+/&*F%&)>
Returns the rst/ust N resuts.
>"".<A
Adds ed vuues rom un externu source.
=/#&3/
Renumes u speced ed, vdcurds cun be used to
specy mutpe eds.
=/A>&!/
Repuces vuues o speced eds vth u speced
new value.
=/_
Speces reguur expresson to use to extruct eds.
$/&=!+
Filters results to those that match the search expres-
sion.
$"=%
Sorts seurch resuts by the speced eds.
$%&%$
lrovdes stutstcs, grouped optonuy by eds.
%"AF=&=/
Dspuys the most/eust common vuues o u ed.
%=&#$&!%)"#
Groups search results into transactions.
Optimizing Searches
The key to fast searching is to limit the data to read from disk to an abso-
ute mnmum und then to ter thut dutu us eury us possbe n the seurch
so that processing is done on the smallest amount of data.
Partition data into separate indexes if youll rarely perform searches across
mutpe types o dutu. lor exumpe, put veb dutu n one ndex und re-
wall data in another.
143
More tips:
- Seurch us speccuy us you cun (C&%&>N/=="=, not P/=="=P).
- Limit the time range (e.g., I5+ not I5().
- lter out unneeded eds us soon us possbe.
- Filter out results as soon as possible before calculations.
- For report generating searches, use the Advanccd Charting view,
and not the TimcIinc view, which calculates timelines.
- Turn off the ficId Discovcry switch when not needed.
- Use summary indexes to precalculate commonly used values.
- Make sure your disk I/O is the fastest you have available.
SEARCH EXAMPLES
fiItcr RcsuIts
Filter results to only include those
with C&)> in their raw text and $%&I
%<$Q4.
c U $/&=!+ C&)> $%&%<$Q4
Remove duplicates of results with the
same host value.
c U */*<A +"$%
Keep only search results whose N=&(
ed contuns ll uddresses n the
nonroutable class A (10.0.0.0/8).
c U =/D/_ N=&(QlXqg}^*Y54E^
*t527uÊ^*t527uÊ^*t527u
Xq}^*Yl
Group RcsuIts
Cluster results together, sort by their
!><$%/=N!"<#% values, and then
return the 20 largest clusters (in data
size).
c U !><$%/= %Q4E:
$+"(!"<#%Q%=)3)%Q64
I!><$%/=N!"<#%
Group results that have the same host
and cookie, occur within 30 seconds
of each other, and do not have a
pause greater than 5 seconds be-
tween each event into a transaction.
c U %=&#$&!%)"# +"$% !"".)/
3&_$A&#Q74$ 3&_A&<$/Q0$
Group results with the same IP ad-
dress (!>)/#%)A) und vhere the rst
result contains signon and the last
result contains purchase.
c U %=&#$&!%)"# !>)/#I
%)A $%&=%$()%+Ql$)D#"#l
/#*$()%+QlA<=!+&$/l
Exploring Splunk
144
Ordcr RcsuIts
Return the rst 20 resuts.
c U +/&* 64
Reverse the order of a result set.
c U =/J/=$/
Sort results by )A value (in ascend-
ing order) and then by <=> value (in
descending order).
c U $"=% )A2 I<=>
Return the last 20 results (in reverse
order).
c U %&)> 64
Rcporting
Return events with uncommon
values.
c U &#"3&>"<$J&></
&!%)"#QC)>%/= A%+=/$+Q4E46
Return the maximum "*/>&8" by
"$)`/", where "$)`/" is broken down
into a maximum of 10 equal sized
buckets.
c U !+&=% 3&_X*/>&8Y B8 $)`/
B)#$Q54
Return 3&_X*/>&8Y for each value of
foo split by the value of B&=.
c U !+&=% 3&_X*/>&8Y "J/= C""
B8 B&=
Return 3&_X*/>&8Y for each value of
C"".
c U !+&=% 3&_X*/>&8Y "J/= C""
Remove all outlying numerical val-
ues.
c U "<%>)/=
Remove duplicates of results with the
same host value and return the total
count of the remaining results.
c U $%&%$ *!X+"$%Y
Return the average for each hour of
uny unque ed thut ends vth the
string >&8 (such as delay, xdelay, and
relay).
c U $%&%$ &JDXP>&8Y B8 *&%/N
+"<=
Calculate the average value of m@K
each minute for each host.
c U %)3/!+&=% $A&#Q53 &JDXm@KY
B8 +"$%
Create a timechart of the count of
from web sources by host.
c U %)3/!+&=% !"<#% B8 +"$%
Return the 20 most common values
of the <=> ed.
c U %"A >)3)%Q64 <=>
Return the least common values of
the <=> ed.
c U =&=/ <=>
145
Add ficIds
Set velocity to distance / time.
c U /J&> J/>"!)%8Q*)$%&#!/F
%)3/
Lxtruct rom und to eds usng
regular expressions. If a raw event
contains p="3- H<$&# G"- L&J)*,
then C="3QH<$&# and %"QL&J)*.
c U =/_ C)/>*QN=&( lp="3-
XqgC="3WEPY G"- Xqg%"WEPYl
Save the running total of count in a
ed cued %"%&>N!"<#%.
c U &!!<3 !"<#% &$ %"%&>N!"<#%
For each event where !"<#% exists,
compute the difference between
!"<#% and its previous value and
store the result in !"<#%*)CC.
c U */>%& !"<#% &$ !"<#%*)CC
fiItcr ficIds
Keep the +"$% and )A eds, und
display them in the order: +"$%, )A.
c U C)/>*$ d +"$%2 )A
Remove the +"$% and )A eds.
c U C)/>*$ I +"$%2 )A
Modify ficIds
leep the host und p eds, und ds-
play them in the order: host, ip.
c U C)/>*$ d +"$%2 )A
Remove the host und p eds.
c U C)/>*$ I +"$%2 )A
MuItivaIucd ficIds
Combine the multiple values of the
=/!)A)/#%$ ed nto one vuue.
c U #"3J =/!)A)/#%$
Separate the values of the
=/!)A)/#%$ ed nto mutpe ed
values, displaying the top recipients.
c U 3&./3J */>)3Ql2l =/!)A)I
/#%$ U %"A =/!)A)/#%$
Create new results for each value of
the mutvuue ed =/!)A)/#%$.
c U 3J/_A&#* =/!)A)/#%$
Combine each result that is identical
except for its S/!"=*Z<3B/=, setting
S/!"=*Z<3B/= to u mutvuued ed
with all the varying values.
c U C)/>*$ RJ/#%m"*/2 m&%/D"I
=82 S/!"=*Z<3B/=
U 3J!"3B)#/ */>)3Q\2\ S/!"=*I
Z<3B/=
Find the number of =/!)A)/#%
values.
c U /J&> %"N!"<#% Q
3J!"<#%X=/!)A)/#%$Y
lnd the rst emu uddress n the
=/!)A)/#% ed.
c U /J&> =/!)A)/#%NC)=$% Q
3J)#*/_X=/!)A)/#%24Y
Exploring Splunk
146
Find all =/!)A)/#% values that end in
.net or .org
c U /J&> #/%"=DN=/!)A)/#%$
Q 3JC)>%/=X3&%!+X=/!)A)/#%2
lÊ#/%jlY TS 3&%!+X=/!)A)/#%2
lÊ"=DjlYY
Find the combination of the values of
C"", lB&=l, and the values of B&`.
c U /J&> #/(J&> Q
3J&AA/#*XC""2 lB&=l2 B&`Y
lnd the ndex o the rst recpent
value that matches lÊ"=Djl
c U /J&> "=D)#*/_ Q
3JC)#*X=/!)A)/#%2 lÊ"=DjlY
lookup TabIcs
Look up the value of each event's
user ed n the ookup tube
<$/=%"D="<A, setting the events
group ed.
c U >"".<A <$/=%"D="<A <$/=
"<%A<% D="<A
Write the search results to the lookup
e <$/=$E!$J.
c U "<%A<%>"".<A <$/=$E!$J
Reud n the ookup e users.csv
as search results.
c U )#A<%>"".<A <$/=$E!$J
EVAL FUNCTIONS
The /J&> command calculates an expression and puts the resulting value
nto u ed (e.g., [EEEU /J&> C"=!/ Q 3&$$ P &!!/>/=&%)"#\). The fol-
lowing table lists the functions /J&> understands, in addition to basic
arithmetic operators (+ - * / %), string concatenation (e.g., ~EEEU /J&>
#&3/ Q >&$% E [2 [ E >&$%z), and Boolean operations (AND OR NOT
XCR < > <= >= != = == LllL).
Iva| Iunc||ons 1ab|e
function Dcscription fxampIcs
&B$XkY
Returns the absolute
value of X.
&B$X#<3B/=Y
!&$/Xk2ll2cY
Takes pairs of argu-
ments X and Y, where X
arguments are Boolean
expressions that, when
evaluated to TRUE, re-
turn the corresponding
Y argument.
!&$/X/=="= QQ M4M2
lZ"% C"<#*l2 /=="=
QQ 0442l?#%/=#&>
H/=J/= R=="=l2 /=I
="= QQ 6442 lTolY
!/)>XkY
Ceiling of a number X.
!/)>X5E:Y
!)*=3&%!+Xlkl2Y
ldentes ll uddresses
that belong to a subnet.
!)*=3&%!+Xl567E576E
76E4F60l2)AY
147
!"&>/$!/Xk2cY
Returns the rst vuue
that is not null.
!"&>/$!/X#<>>XY2
lS/%<=#/* J&>l2
#<>>XYY
/_&!%XkY
Evaluates an expres-
sion X using double
precson outng pont
arithmetic.
/_&!%X7E5MP#<3Y
/_AXkY
Returns eX.
/_AX7Y
C>""=XkY
Returns the oor o u
number X.
C>""=X5E:Y
)CXk22yY
If X evaluates to TRUE,
the result is the second
argument Y. If X evalu-
ates to FALSE, the result
evaluates to the third
argument Z.
)CX/=="=QQ6442
lTol2 lR=="=lY
)$B"">XkY
Returns TRUE if X is
Boolean.
)$B"">XC)/>*Y
)$)#%XkY
Returns TRUE if X is an
integer.
)$)#%XC)/>*Y
)$#"%#<>>XkY
Returns TRUE if X is not
NULL.
)$#"%#<>>XC)/>*Y
)$#<>>XkY
Returns TRUE if X is
NULL.
)$#<>>XC)/>*Y
)$#<3XkY
Returns TRUE if X is a
number.
)$#<3XC)/>*Y
)$$%=XY
Returns TRUE if X is a
string.
)$$%=XC)/>*Y
>/#XkY
This function returns the
character length of a
string X.
>/#XC)/>*Y
>)./Xk2llY
Returns TRUE if and
only if X is like the
SQLite pattern in Y.
>)./XC)/>*2 lC""flY
>#XkY
Returns the natural log
of X.
>#XB8%/$Y
>"DXk2Y
Returns the log of the
rst urgument X usng
the second argument Y
as the base. Y defaults
to 10.
>"DX#<3B/=26Y
Exploring Splunk
148
>"(/=XkY
Returns the lowercase
of X.
>"(/=X<$/=#&3/Y
>%=)3Xk2Y
Returns X with the char-
acters in Y trimmed from
the left side. Y defaults
to spaces and tabs.
>%=)3Xl yyy&B!yy l2
l ylY
3&%!+Xk2Y
Returns True, if X match-
es the regex pattern Y.
3&%!+XC)/>*2
l^*t527uÊ^*jlY
3&_Xk2cY
Returns the greater of
the two values.
3&_X*/>&82 38*/>&8Y
3*0XkY
Returns the MD5 hash
of string value X.
3*0XC)/>*Y
3)#Xk2cY
Returns the min.
3)#X*/>&82 38*/>&8Y
3J!"<#%XkY
Returns the number of
values of X.
3J!"<#%X3<>%)C)/>*Y
3JC)>%/=XkY
Filters a multivalued
ed bused on the oo-
ean expression X.
3JC)>%/=X3&%!+X/3&)
>2 l#/%jlYY
3J)#*/_Xk22yY
Returns a subset of the
mutvuued ed X rom
start position (zero-
based) Y to Z (optional).
3J)#*/_X 3<>%)I
C)/>*2 6Y
3Je")#Xk2Y
Given a multivalued
ed X und strng dem-
iter Y, joins the individu-
al values of X using Y.
3Je")#XC""2 lslY
#"(XY
Returns the current
time, represented in
Unix time.
#"(XY
#<>>XY
Takes no arguments and
returns NULL.
#<>>XY
#<>>)CXk2Y
Given two arguments,
eds X und Y, returns
X if the arguments are
different; returns NULL,
otherwise.
#<>>)CXC)/>*,2
C)/>*rY
A)XY
Returns the constant pi.
A)XY
A"(Xk2Y
Returns XY.
A"(X6254Y
149
=&#*"3XY
Returns a pseudo-
random number ranging
from 0 to 2147483647.
=&#*"3XY
=/>&%)J/N%)3/Xk2Y
Given epochtime time
X and relative time
specer Y, returns the
epochtime value of Y
applied to X.
=/>&%)J/N
%)3/X#"(XY2lI5*x*lY
=/A>&!/Xk22yY
Returns a string formed
by substituting string Z
for every occurrence of
regex string Y in string
X.
S/%<=#$ *&%/ ()%+
%+/ 3"#%+ &#* *&8
#<3B/=$ $()%!+/*2
$" )C %+/ )#I
A<% )$ 5F56F644:
%+/ =/%<=# J&></
)$ 56F5F644:-
=/A>&!/X*&%/2
lX^*t526uYF
X^*t526uYFl2
l^6F^5FlY
="<#*Xk2Y
Returns X rounded to
the amount of decimal
puces speced by Y.
The default is to round
to an integer.
="<#*X7E0Y
=%=)3Xk2Y
the right side. If Y is not
speced, spuces und
tabs are trimmed.
=%=)3Xl yyyy&B!yy
l2 l ylY
$/&=!+3&%!+XkY
Returns true if the event
matches the search
string X.
$/&=!+3&%!+XlC""
,ZL B&=lY
$A>)%Xk2llY
Returns X as a
mutvuued ed, spt
by delimiter Y.
$A>)%XC""2 lslY
$v=%XkY
Returns the square root
of X.
$v=%X:Y
$%=C%)3/Xk2Y
Returns epochtime
value X rendered using
the ormut speced by
Y.
$%=C%)3/XN%)3/2
lf|-fOlY
Exploring Splunk
150
$%=A%)3/Xk2Y
Given a time represent-
ed by a string X, returns
value parsed from
format Y.
$%=A%)3/X%)3/H%=2
lf|-fOlY
$<B$%=Xk22yY
Returns u substrng ed
X from start position
(1-based) Y for Z (op-
tional) characters.
$<B$%=Xl$%=)#Dl2
52 7Y
d$<B$%=Xl$%=)#Dl2
I7Y
%)3/XY
Returns the wall-clock
time with microsecond
resolution.
%)3/XY
%"#<3B/=Xk2Y
Converts input string X
to a number, where Y
(optional, defaults to 10)
denes the buse o the
number to convert to.
tonumber("0A4",16)
%"$%=)#DXk2Y
Returns u ed vuue o
X as a string. If X is a
number, it reformats it
as a string; if a Boolean
value, either "True" or
"False". If X is a number,
the second argument Y
is optional and can ei-
ther be "hex" (convert X
to hexadecimal), "com-
mas" (formats X with
commas and 2 decimal
places), or "duration"
(converts seconds X to
readable time format
HH:MM:SS).
This example re-
turns foo=615 and
foo2=00:10:15:
c U /J&> C""Q;50
U/J&>
C""6Q%"$%=)#DXC""2\
*<=&%)"#\Y
%=)3Xk2Y
both sides. If Y is not
speced, spuces und
tabs are trimmed.
trim(" ZZZZabcZZ ", "
Z")
%8A/"CXkY
Returns a string repre-
sentation of its type.
This example returns:
"NumberStringBoolIn-
valid":
%8A/"CX56Yd
%8A/"CX[$%=)#D\Yd
%8A/"CX5QQ6Yd
%8A/"CXB&*C)/>*Y
151
<AA/=XkY
Returns the uppercase
of X.
<AA/=X<$/=#&3/Y
<=>*/!"*/XkY
Returns the URL X
decoded.
<=>*/!"*/X[+%%Af7,f
6pf6p(((E$A><#.E
!"3f6p*"(#>"&*f7p=f
7L+/&*/=\Y
validate(X,Y,) Given pairs of argu-
ments, Boolean expres-
sions X and strings
Y, returns the string Y
corresponding to the
rst expresson X thut
evaluates to False and
defaults to NULL if all
are True.
validate(isint(port),
"ERROR: Port is not
un nteger", port >= l
AND port <= 65535,
"ERROR: Port is out of
range")
COMMON STATS FUNCTIONS
Common statistical functions used with the !+&=%, $%&%$, and %)3/!+&=%
commands. Field names can be wildcarded, so &JDXP*/>&8Y might cal-
culate the average of the */>&8 and _*/>&8 eds.
function Dcscription
&JDXkY
Returns the uveruge o the vuues o ed X.
!"<#%XkY
Returns the number o occurrences o the ed X. 1o nd-
cute u ed vuue to mutch, ormut X us evu(ed="vuue").
*!XkY
Returns the count o dstnct vuues o the ed X.
C)=$%XkY
Returns the rst seen vuue o the ed X. ln generu, the rst
seen vuue o the ed s the chronoogcuy most recent
nstunce o ed.
>&$%XkY
Returns the ust seen vuue o the ed X.
>)$%XkY
Returns the st o u vuues o the ed X us u mutvuue
entry. 1he order o the vuues reects the order o nput
events.
3&_XkY
Returns the muxmum vuue o the ed X. l the vuues o
X are non-numeric, the max is found from lexicographic
ordering.
3/*)&#XkY
Returns the mdde-most vuue o the ed X.
3)#XkY
Returns the mnmum vuue o the ed X. l the vuues o
X are non-numeric, the min is found from lexicographic
ordering.
3"*/XkY
Returns the most requent vuue o the ed X.
Exploring Splunk
152
A/=!gkWXY
Returns the X-th percente vuue o the ed Y. lor exumpe,
perc5(totu) returns the 5th percente vuue o u ed totu..
=&#D/XkY
Returns the difference between the max and min values of
the ed X.
$%*/JXkY
Returns the sumpe stundurd devuton o the ed X.
$%*/JAXkY
Returns the popuuton stundurd devuton o the ed X.
$<3XkY
Returns the sum o the vuues o the ed X.
$<3$vXkY
Returns the sum o the squures o the vuues o the ed X.
J&></$XkY
Returns the st o u dstnct vuues o the ed X us u
multivalue entry. The order of the values is lexicographical.
J&=XkY
Returns the sumpe vurunce o the ed X.
REGULAR EXPRESSIONS
Regular expressions are useful in many areas, including search com-
mands =/D/_ and =/_; /J&> functions 3&%!+XY and r/A>&!/XY; and in
ed extructon.
REGEX NOTE fXAMPlf fXPlANATlON
^$
white space
^*^$^*
digit space digit
^H
not white space
^*^H^*
digit non-
whitespace digit
^*
Digit
^*^*^*I^*^*I^*^*^
*^*
SSN
^L
not digit
^L^L^L
three non-digits
^(
word character
(letter, number,
or _ )
^(^(^(
three word chars
^{
not a word char-
acter
^{^{^{
three non-word
chars
aEEEb
any included
character
a&I`4I:]b
any char that is a
thru z, 0 thru 9,
or r
aEEEb
no included char-
acter
a_8_b
any char but x, y,
or z
P
zero or more
^(P
zero or more
words chars
d
one or more
^*d
integer
153
q
zero of one
^*^*^*Iq^*^*Iq^*^
*^*^*
SSN with dashes
being optional
U
Or
^(U^*
word or digit char-
acter
Xq@gJ&=W
EEEY
named extraction
Xq@g$$#W^*^*^*I
^*^*Î^*^*^*^*Y
pull out a SSN and
ussgn to 'ssn' ed
Xq- EEE Y
logical grouping
Xq-^(U^*YUXq-^
*U^(Y
word-char then
digit OR digit then
word-char
start of line
^*d
line begins with at
least one digit
j
end of line
^*dj
line ends with at
least one digit
tEEEu
number of repeti-
tions
^*t720u
between 3-5 digits
^
Escape
â
escape the [ char
XqQ EEEY
Lookahead
XqQ^LY/=="=
/=="= must be
preceded by a
non-digit
Xq} EEEY
negative looka-
head
Xq}^*Y/=="=
/=="= cannot be
preceded by a
digit
COMMON SPLUNK STRPTIME FUNCTIONS
strptime formats are useful for /J&> functions $%=C%)3/XY and $%=A%)3/XY
and for timestamping event data.
TlMf
f|
24 hour (leading zeros) (00 to 23)
f?
12 hour (leading zeros) (01 to 12)
fO
Minute (00 to 59)
fH
Second (00 to 61)
fZ
subseconds with width (%3N = millisecs,
%6N = microsecs, %9N = nanosecs)
fA
AM or PM
fy G)3/ `"#/ XiOGY
f$
Seconds since 1/1/1970 (1308677092)
Exploring Splunk
154
DAYS
f*
Day of month (leading zeros) (01 to 31)
fe
Day of year (001 to 366)
f(
Weekday (0 to 6)
f&
Abbreviated weekday (Sun)
f,
Weekday (Sunday)
fB
Abbreviated month name (Jan)
fr
Month name (January)
f3
Month number (01 to 12)
f8
Year without century (00 to 99)
f
Year (2008)
fIf3If*
1998-12-31
f8If3If*
98-12-31
fB f*2 f
Jan 24, 2003
fr f*2 f
January 24, 2003
vUf* fB f8 Q
fIf3If*
q|25 Feb '03 = 2003-02-25|

Exploring Splunk

Загружено:

Сведения о документе

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

Exploring Splunk

Загружено:

Авторское право:

Доступные форматы

Exploring Splunk

SEARCH PROCESSING LANGUAGE (SPL)

2012 by Splunk Inc.

command or a custom script that

, JMS, and AquaLogic

revus og usng the ClSLC Log Lxport

Вам также может понравиться