Splunk Enterprise Metrics 7.0.

0
Generated: 11/17/2017 3:57 pm

Copyright (c) 2017 Splunk Inc. All Rights Reserved

 Table of Contents
Introduction to metrics........................................................................................1
Overview of metrics.....................................................................................1
Get started with metrics..............................................................................2

Get metrics data in...............................................................................................6

Get metrics in from StatsD..........................................................................6
Get metrics in from collectd.......................................................................12
Get metrics in from other sources.............................................................17

Work with metrics..............................................................................................23

Search and monitor metrics......................................................................23
Metrics indexing performance...................................................................25
Best practices for metrics..........................................................................26

i
Introduction to metrics

Overview of metrics
Metrics is a feature for system administrators and IT tools engineers that focuses
on collecting, investigating, monitoring, and sharing metrics from your technology
infrastructure, security systems, and business applications in real time.

Metrics in the Splunk platform uses a custom index type that is optimized for
metric storage and retrieval. To work with metrics, the mstats search command is
included for you to apply numeric aggregations (such as avg, sum, percentile,
and so forth) to isolate and correlate problems from different data sources.

What is a metric?

A metric is a set of measurements containing a timestamp, a metric name, a

value, and a dimension.

timestamp
Indicates when a metric measurement was taken.
metric name
Uses a dotted hierarchy to refer to a namespace (for example,
nginx.upstream.responses.5xx). Any string can be used as metric name.
We recommend that metric names only include lowercase letters,
numbers, underscores, and dots. Dots are used to separate segments of
the namespace to create a hierarchy of metrics.
value
A numeric data point representing the value of a metric such as a count,
or calculated value for a specific time resolution such as a percentile for a
response time metric for the last minute.
dimensions
Provide metadata about the metric. For example:
Region: us-east-1, us-west-1, us-west-2, us-central1
InstanceTypes: t2.medium, t2.large, m3.large, n1-highcpu-2
Technology: nginx, redis, tomcat

You can think of a metric name as something that you are measuring, while
dimensions are categories by which you can filter or group the results.

The following are examples of systems that generate metrics:

1
 IT infrastructure, such as hosts, networks, and devices
System components, such as web servers and databases
Application-specific metrics, such as timers that measure performance of
a function
SaaS
Sensors, such as IoT

What is metrics in the Splunk platform?

Metrics is a feature of the Splunk platform. Metrics includes:

A metric collection framework used to collect and ingest high-volume

metric measurements, typically using a dotted notation for the metric
name and metric hierarchy, such as
collectd.host.docker_stats.app.task.cpu.system, from agents and APIs.
A method to support the collection of existing line metric protocols such as
collectd, StatsD, and DogStatsD.
A framework that can be used by the universal forwarder (UF) and heavy
weight forwarder (HWF) to collect metrics and to optimally and securely
forward the metric payload to a standalone or cluster of metric indexes.
A metric ingestion pipeline that supports the ability to apply props and
transforms at indexing time for well-structured metric payload protocols.
A data catalog that is specific to metrics, providing a way to browse and
filter a large set of metric names and dimensions.

For examples of metrics data in the Splunk platform, see Get metrics data in.

Get started with metrics

The Splunk platform gathers metrics from different sources and stores this data
into a new type of index that is optimized for ingestion and retrieval of metrics.

The Splunk platform supports the following metrics-gathering tools natively:

The collectd agent, a Unix-based daemon, with the write_HTTP plugin.

Collectd supports over 100 front-end plugins.
The StatsD line protocol, which is used by a wide range of client libraries
and other open source tools.

Both of these tools are light weight and easy to use, and they have a large

2
community of support. If you want to start gathering performance metrics from
your applications and systems, review these tools to determine whether either of
them suits your environment.

If you prefer to use a different metrics-gathering tool, you can still use the Splunk
platform to collect and analyze your data with manual configuration.

Metrics data format

Metrics data uses a specific format with the following fields.

Writeable
Field Required or Description Example
Internal
metric_name X Writeable The metric name. os.cpu.user
The timestamp of the
2017-08-14
_time X Writeable metric in UNIX time
17:12:39.000
notation.
The numeric value of the
metric. This field is a
64-bit floating point
_value X Writeable 42.12345
number, which supports
precision between 15
and 17 decimal digits.
<dimension 0> An arbitrary number of
... X Writeable fields, indicating how ip
<dimension n> metrics can be split.
An auto-generated
internal field that
contains the names of
all of the dimensions in
_dims X Internal the metric event. The _dims::ip
purpose of this field is to
return a list of unique
dimension names in a
metrics index.
The type of metric. Only
metric_type Writeable g
"g" (gauge) is supported.
source Internal The source of the udp:8125

3
 metrics data.
The origin host. A
host X Internal standard field in Splunk server007
software.
The metrics index name.
index X Internal A standard field in metricsindex
Splunk software.
The data structure of the
sourcetype X Internal metric. A standard field statsd
in Splunk software.
Supported line protocols

Metrics in the Splunk platform natively supports the following metric line
protocols:

Plain StatsD over UDP/TCP

The StatsD extension with dimensions over UDP/TCP
Collectd over HTTPS using HTTP Event Collector (HEC)

For details about getting data in, see Get metrics in from StatsD and Get metrics
in from collectd.

To support other line metric protocols, you can use custom transformations to get
metrics data into Splunk platform from other tools. For details, see Get metrics
from other clients.

Metrics source types

The Splunk platform includes the following pre-trained source types to support
the most widely-supported line metric protocols:

4
 Source type
Description
name
Supports data using the metric line protocols for plain StatsD
statsd
and the StatsD extension with dimensions.
collectd_http Supports data using the metric line protocol for collectd.
Supports data in CSV format. For usage details, see Get
metrics_csv
metrics in from other sources.
Metrics index

To store and analyze metrics data as efficiently as possible, metrics data is

stored in a new type of index just for metrics. A metrics index can be used only
for metrics data. You cannot convert an events index to a metrics index, or vice
versa.

There is no default metrics index. Summary indexing does not work with metrics.

To learn more, see Create metrics indexes in the Managing Indexers and
Clusters of Indexers manual.

For information about how metrics data is metered, see How Splunk Enterprise
licensing works in the Admin Manual.

Search and CLI commands with metrics

To analyze data and enumerate items in a metrics index, use the mstats and
mcatalog search commands. Other search commands that work with events do
not work with metrics. For example, the delete command does not work with
metrics. For more about searching a metrics index, see Search and monitor
metrics.

Administrative CLI commands may not all work with metrics. You can use
commands such as add index and list index with metrics when using the
-datatype metric parameter. See Create metrics indexes in the Managing
Indexers and Clusters of Indexers manual.

5
Get metrics data in

Get metrics in from StatsD

StatsD is a network daemon that runs on the Node.js platform, sending metrics
over UDP or TCP. For an overview of StatsD, see Measure Anything, Measure
Everything on the Code as Craft website.

StatsD has several implementations, some of which encode dimensions in

different ways. The Splunk platform supports the following formats natively:

Plain StatsD line metric protocol:

<metric_name>:<_value>|<metric_type>

Example metric:

performance.os.disk:1099511627776|g

StatsD line metric protocol with dimensions extension:

<metric_name>:<_value>|<metric_type>|#dim1:valueX,dim2:valueY

Example metric:

performance.os.disk:1099511627776|g|#region:us-west-1,datacenter:us-west-1a,rack:6
path:/dev/sdal,fstype:ext3

For more about formats for metric names and dimensions, see Best practices for
metrics.

Using other StatsD formats

If you use a StatsD implementation that uses a different format for dimensions
from the ones that the Splunk platform supports natively, for example one that
embeds dimensions within the metric name, you can still use metrics in the
Splunk platform. However, you'll need to customize Splunk configuration files to
specify how to extract dimensions from your format.

Another option is to use StatsD to gather metrics, but use collectd to send the
data to the Splunk platform over HTTP. The benefit of this method is that collectd

6
normalizes the dimension format in the metrics data. For more, see Get metrics
in from collectd.

Set up a data input for StatsD data

After you configure your data source to send data in the StatsD protocol, create a
UDP or TCP data input in the Splunk platform to listen for StatsD data on an
open port.

1. In Splunk Web, go to Settings > Data inputs.

2. Under Local inputs, click Add new next to UDP or TCP, depending on
the type of input you want to create.

When using UDP ports to ingest metric data, you cannot use parallel
ingestion or the multiple pipeline sets feature.

3. For Port, enter the number of the port you are using for StatsD.
4. Click Next.
5. Click Select Source Type, then select Metrics > statsd.
6. For Index, select an existing metrics index. Or, click Create a new index
to create one.
If you choose to create an index, in the New Index dialog box:
1. Enter an Index Name. User-defined index names must consist of
only numbers, lowercase letters, underscores, and hyphens. Index
names cannot begin with an underscore or hyphen.
2. For Index Data Type, click Metrics.
3. Configure additional index properties as needed.
4. Click Save.
7. Click Review, then click Submit.

Extract StatsD dimensions for unsupported formats

Many StatsD clients embed dimension names in the metric name. For example,
let's say your StatsD client uses the following line metric protocol format, which is
not supported natively by the Splunk platform:

<dimension>.<metric_name>:<value>|<metric_type>

Here's an example of a metric returned using this unsupported format:

10.1.1.198.cpu.percent:75|g

The metric field values should be:

7
 metric_name=cpu.percent
_value=75
metric_type=g

The extracted dimension should be:

ip=10.1.1.198

To create the correct results, you must edit Splunk configuration files or use the
REST API to create a custom source type that specifies how to extract
dimensions from this metrics data.

Configure dimension extraction by editing configuration files

1. Define a custom source type for your StatsD metrics data.

1. In a text editor, open the props.conf configuration file from the local
directory for the location you want to use, such as the Search &
Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or from
the system ($SPLUNK_HOME/etc/system/local). If a props.conf file
does not exist in this location, create a text file and save it to that
location.
2. Append a stanza to the props.conf file as follows:

# props.conf

[<metrics_sourcetype_name>]
METRICS_PROTOCOL = statsd
STATSD-DIM-TRANSFORMS =
<statsd_dim_stanza_name1>,<statsd_dim_stanza_name2>...
metrics_sourcetype_name: The name of your custom
metrics source type.
statsd_dim_stanza_name: A comma-separated list of
transforms stanza names that specify how to extract
dimensions. If only one stanza is used for the source type,
and if the transforms stanza name is same as the
metrics_sourcetype_name, this
STATSD-DIM-TRANSFORMS setting can be omitted.
2. Define one or more regular expressions to extract the dimensions from
metric_name.
1. In a text editor, open the transforms.conf configuration file from the
local directory for the location you want to use, such as the Search
& Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or from
the system ($SPLUNK_HOME/etc/system/local). If a
transforms.conf file does not exist in this location, create a text file
and save it to that location.

8
 2. Append a stanza for each regular expression as follows:

# transforms.conf

[statsd-dims:<unique_transforms_stanza_name>]
REGEX = <regular expression>
REMOVE_DIMS_FROM_METRIC_NAME = <Boolean>
unique_transforms_stanza_name: A unique name for this
stanza.
REGEX = <regular expression>: A regular expression that
defines how to match and extract dimensions from StatsD
metrics data. The Splunk platform supports a named
capturing-group extraction format
(?<dim1>group)(?<dim2>group)... to provide dimension
names for the corresponding values that are extracted.
REMOVE_DIMS_FROM_METRIC_NAME = <Boolean>:
Specifies whether unmatched segments of the StatsD dotted
name segment are used as the metric_name.

When true, dimension values are removed from the

measurement and the unmatched portion becomes the
metric_name. The default value is true.

When false, extracted dimension values are included in the

metric_name.

For example, a metric measurement name is "x.y.z". The

regular expression matches "y" and "z". When
REMOVE_DIMS_FROM_METRIC_NAME is true,
metric_name is "x". When false, metric_name is "x.y.z".
3. Create a data input for this source type as described in Set up a data input
for StatsD data, and select your custom source type.

For more about editing these configuration files, see About configuration files,
props.conf, and transforms.conf in the Admin Manual.

Examples of configuring dimension extraction

Let's say you have StatsD metrics data such as:

data=mem.percent.used.10.2.3.4.windows:33|g

You need to extract the "ipv4" and "os" dimensions.

9
If you defined two regular expressions, one for "ipv4" and one for "os", you would
append the following stanzas to your configuration files:

# props.conf.example

[my_custom_metrics_sourcetype]
METRICS_PROTOCOL = statsd
STATSD-DIM-TRANSFORMS = regex_stanza1, regex_stanza2

# transforms.conf.example

[statsd-dims:regex_stanza1]
REGEX = (?<ipv4>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})
REMOVE_DIMS_FROM_METRIC_NAME = true
[statsd-dims:regex_stanza2]
REGEX = \S+\.(?<os>\w+):
REMOVE_DIMS_FROM_METRIC_NAME = true
Now let's say you can accomplish this same extraction using a single regular
expression. In this case, you would append the following stanzas to your
configuration files:

# props.conf.example

[my_custom_metrics_sourcetype]
METRICS_PROTOCOL = statsd

# transforms.conf.example

[statsd-dims:my_custom_metrics_sourcetype]
REGEX = (?<ipv4>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\.(?<os>\w+):
REMOVE_DIMS_FROM_METRIC_NAME = true
Notice that the STATSD-DIM-TRANSFORMS setting in the props.conf
configuration file is not needed when only a single regular expression is used for
a source type.

Configure dimension extraction for StatsD by using the REST API

1. Define a custom source type for your StatsD metrics data by using the
/services/saved/sourcetypes REST endpoint:
https://<host>:<mPort>/services/saved/sourcetypes \
-d
"name=<metrics_sourcetype_name>&METRICS_PROTOCOL=statsd&STATSD-DIM-TRANSFORMS=<sta
metrics_sourcetype_name: The name of your custom metrics
source type.
statsd_dim_stanza_name: A list of transforms stanza names that
specify how to extract dimensions. If only one stanza is used for the

10
 source type, and if the transforms stanza name is same as the
metrics_sourcetype_name, this STATSD-DIM-TRANSFORMS
setting can be omitted.
For example, enter the following command:

curl -k -u admin:changeme
https://localhost:8089/services/saved/sourcetypes \
-d
"name=statsd_custom&METRICS_PROTOCOL=statsd&STATSD-DIM-TRANSFORMS=statsd-ex&SHOULD
2. Create one or more regular expressions to extract the dimensions from
metric_name by using the /data/transforms/statsdextractions REST
endpoint:

https://<host>:<mPort>/services/data/transforms/statsdextractions
\
-d "name=<unique_transforms_stanza_name>&REGEX=<regular
expression>&REMOVE_DIMS_FROM_METRIC_NAME=<Boolean>"
unique_transforms_stanza_name: A unique name for this stanza.
REGEX = <regular expression>: A regular expression that defines
how to match and extract dimensions from StatsD metrics data.
The Splunk platform supports a named capturing-group extraction
format (?<dim1>group)(?<dim2>group)... to provide dimension
names for the corresponding values that are extracted.
REMOVE_DIMS_FROM_METRIC_NAME = <Boolean>: Specifies
whether unmatched segments of the StatsD dotted name segment
are used as the metric_name.

When true, dimension values are be removed from the

measurement and the unmatched portion becomes the
metric_name. The default value is true.

When false, extracted dimension values are included in the

metric_name.

For example, a metric measurement name is "x.y.z". The regular

expression matches "y" and "z". When
REMOVE_DIMS_FROM_METRIC_NAME is true, metric_name is
"x". When false, metric_name is "x.y.z".

For example, enter the following command:

curl -k -u admin:changeme
https://localhost:8089/services/data/transforms/statsdextractions

11
 \
-d
"name=statsd-ex&REGEX=\.(?<hostname>\S%2B?)\.(?<ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1
3. Reload the metrics processor to load the configuration changes by using
the /admin/metrics-reload/_reload REST endpoint:

https://<host>:<mPort>/services/admin/metrics-reload/_reload
For example, enter the following command:

curl -k -u admin:changeme \
https://localhost:8089/services/admin/metrics-reload/_reload
4. Create a data input for this source type as described in Set up a data input
for StatsD data, and select your custom source type.

For more about using the Splunk REST API, see Using the REST API reference,
/data/transforms/statsdextractions, and /admin/metrics-reload/_reload in the
REST API Reference Manual.

Get metrics in from collectd

Collectd is an open source daemon that collects performance metrics from a
variety of sources. Using the collectd write_http plugin, collectd sends metrics
data to a data input in the Splunk platform using the HTTP Event Collector
(HEC).

To send metrics using collectd, do the following:

1. Configure the HTTP Event Collector (HEC) data input.

2. Install collectd.
3. Configure collectd.
4. Start collectd.

Configure the HTTP Event Collector (HEC) data input

The HTTP Event Collector (HEC) is an endpoint that lets you send application
events to your deployment of the Splunk platform using the HTTP or Secure
HTTP (HTTPS) protocols. Configure this data input before setting up collectd
because you'll need to use data input details for the collectd configuration.

1. In Splunk Web, click Settings > Data Inputs.

2. Under Local Inputs, click HTTP Event Collector.
3. Verify that HEC is enabled.

12
 1. Click Global Settings.
2. For All Tokens, click Enabled if this button is not already selected.
3. Note the value for HTTP Port Number, which you'll need to
configure collectd.
4. Click Save.
4. Configure an HEC token for sending data by clicking New Token.
5. On the Select Source page, for Name, enter a token name, for example
"collectd token".
6. Leave the other options blank or unselected.
7. Click Next.
8. On the Input Settings page, for Source type, click Select.
9. Click Select Source Type, then select Metrics > collectd_http.
10. Next to Default Index, select your metrics index, or click Create a new
index to create one.
If you choose to create an index, in the New Index dialog box:
1. Enter an Index Name. User-defined index names must consist of
only numbers, lowercase letters, underscores, and hyphens. Index
names cannot begin with an underscore or hyphen.
2. For Index Data Type, click Metrics.
3. Configure additional index properties as needed.
4. Click Save.
11. Click Review, and then click Submit.
12. Copy the Token Value that is displayed, which you'll need to configure
collectd.

Add collectd events directly to a metrics index

To test your data input, you can send collectd events directly to your metrics
index using the /collector/raw REST API endpoint, which accepts data in the
collectd JSON format. Your metrics index is assigned to an HEC data input that
has its unique HEC token, and "collectd_http" as its source type.

The following example shows a curl command that sends a collectd event to the
index associated with your HEC token:

curl -k
https://localhost:8088/services/collector/raw?sourcetype=collectd_http
\
-H "Authorization: Splunk <HEC_token>"
\
-d
'[{"values":[164.9196798931339196],"dstypes":["derive"],"dsnames":["value"],"time":15053

13
You can verify the HEC data input is working by running a search using mcatalog
to list all metric names, with the time range set to ?All Time?, for example:

| mcatalog values(metric_name) WHERE index=<your_metrics_index> AND

metric_name=protocols.protocol_counter.InOctets.value

Or, use the Metrics Catalog REST endpoint to list metric names:

curl -k -u <admin:passwd>
"https://localhost:8089/services/catalog/metricstore/metrics?earliest=0"
See Getting data in with HTTP Event Collector and Event formatting on the
Splunk Developer Portal. See mstats and mcatalog in the Search Reference
manual. See Metrics Catalog endpoint descriptions, /collector, /collector/raw in
the REST API Reference Manual.

Install collectd

Install the collectd agent on the computers in your system from which you want to
collect metrics.

1. Go to the First steps page on the collectd website.

2. Install collectd version 5.6 or higher, following the installation instructions
for your operating system.

Configure collectd

The collectd server is an optional daemon that can be used to aggregate metrics
from different inputs and one-to-many collectd clients.

Configure the collectd client to collect data by configuring plugins in the

collectd.conf configuration file. The location of the collectd.conf file depends on
your operating system. For details, see "Configuration" on the First steps page
on the collectd website.

The write_http plugin

The write_http plugin requires the following fields from your HEC data input:

Field
Description Syntax
name
URL URL to URL URL
which the "https://<Splunk_host>:<HTTP_port>/services/collector/raw" "https://10.66.

14
 values are
submitted.
This URL
includes
your Splunk
host
machine (IP
address,
host name,
or load
balancer
name), and
the HTTP
port
number.
An HTTP
header to Header "Auth
Header Header "Authorization: Splunk <HEC_token>"
add to the b0221cd8-c4b
request.
The format
Format Format "JSON" Format "JSON
of the data.
Enable and configure plugins

Enable each plugin below by uncommenting the plugin's LoadPlugin statement,

then configure the plugin as described. Most of these plugins are for gathering
basic OS-level metrics. The logfile plugin is needed for debugging purposes.
You can configure additional plugins according to your requirements.

You might need to install some plugins separately, depending on your installation
method and operating system. For details, see the collectd website.

Plugin Suggested configuration

LoadPlugin cpu
<Plugin cpu>
cpu ReportByCpu true
</Plugin>
LoadPlugin interface
interface
Use the default configuration.
load LoadPlugin load
<Plugin load>

15
 ReportRelative true
</Plugin>
LoadPlugin logfile
<Plugin logfile>
LogLevel info
logfile File STDOUT
Timestamp true
PrintSeverity false
</Plugin>
LoadPlugin memory
<Plugin memory>
memory ValuesAbsolute true
ValuesPercentage true
</Plugin>
LoadPlugin network

network Enable this plugin only if the collectd client is not on the same
machine as the connectd server, then use the default configuration.
LoadPlugin syslog
syslog
Use the default configuration.
You need the values from your HEC data input to configure this
plugin. LoadPlugin write_http
<Plugin write_http>
<Node "node1">
URL
"https://<Splunk_host>:<HTTP_port>/services/collector/raw"
write_http Header "Authorization: Splunk <HEC_token>"
Format "JSON"
VerifyPeer false
VerifyHost false
Metrics true
StoreRates true
</Node>
</Plugin>
Start collectd

To start collectd, follow the instructions under "Starting the daemon" on the First
steps page on the collectd website.

Modules for all of the enabled plugins in your collectd.conf file must be installed.
Errors are displayed for any modules that are missing. For more about the
available collectd plugins, see Table of Plugins on the collectd Wiki website.

16
Install modules according to your operating system. For example, on Linux you
must install collectd-write_http.x86_64 to use the write_http plugin.

Tips:

For troubleshooting, refer to the collectd log file enabled by the logfile
plugin for details.
Use the File setting in the logfile plugin to write to a specified file rather
than to standard output. For example:

<Plugin logfile>
LogLevel info
File "/var/log/collectd.log"
Timestamp true
PrintSeverity false
</Plugin>
If you are installing collectd on Linux, you can use yum to list available
modules. For example, use this CLI command:
yum list | grep collectd
In the collectd.conf file, set the FQDNLookup setting to false to render a
friendly name for the domain name.

Get metrics in from other sources

If you are gathering metrics from a source that is not natively supported, you can
still add this metrics data to a metrics index.

Get metrics in from files in CSV format

If your metrics data is in CSV format, use the metrics_csv pre-trained source
type. Your CSV data must contain the following fields.

Field name Required Description Example

Epoch time (elapsed time since
metric_timestamp X 1504907933.000
1/1/1970), in milliseconds.
The metric name using
metric_name X os.cpu.percent
dotted-string notation.
_value X A numerical value. 42.12345
All other fields are treated as
dimensions ip
dimensions.

17
To add CSV data to a metrics index, create a data input with the following:

Source type: Metrics > metrics_csv

Index: a metrics index

See Monitor files and directories in the Getting Data In manual, and Create
metrics indexes in the Managing Indexers and Clusters of Indexers manual.

Get metrics in from clients over TCP/UDP

You can add metrics data from a client that is not natively supported to a metrics
index by manually configuring a source type for your data, then defining regular
expressions to specify how the Splunk software should extract the required
metrics fields. See Metrics data format.

For example, let's say you are using Graphite. The Graphite plaintext protocol
format is:

<metric path> <metric value> <metric timestamp>

A sample metric might be:

510fcbb8f755.sda2.diskio.read_time 250 1487747370

To index these metrics, edit Splunk configuration files to manually specify how to
extract fields.

Configure field extraction by editing configuration files

1. Define a custom source type for your metrics data.

1. In a text editor, open the props.conf configuration file from the local
directory for the location you want to use, such as the Search &
Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or the
system ($SPLUNK_HOME/etc/system/local). If a props.conf file
does not exist in this location, create a text file and save it to that
location.
2. Append a stanza to the props.conf file as follows:

# props.conf

[<metrics_sourcetype_name>]
TIME_PREFIX = <regular expression>
TIME_FORMAT = <strptime-style format>

18
 TRANSFORMS-<class> = <transform_stanza_name>
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
pulldown_type = 1
category = Metrics
metrics_sourcetype_name Name of your custom metrics
source type.
TIME_PREFIX = regular expression: A regular expression
that indicates where the timestamp is located.
TIME_FORMAT = strptime-style format: A strptime format
string used to extract the date. For more about strptime, see
Configure timestamp recognition in the Getting Data In
manual.
TRANSFORMS-<class> = <transform_stanza_name>: class
is a unique literal string that identifies the namespace of the
field to extract. transform_stanza_name is the name of the
name of your stanza in transforms.conf that indicates how to
extract the field.
2. Define a regular expression for each metrics field to extract.
1. In a text editor, open the transforms.conf configuration file from the
local directory for the location you want to use, such as the Search
& Reporting app ($SPLUNK_HOME/etc/apps/search/local/) or the
system ($SPLUNK_HOME/etc/system/local). If a transforms.conf
file does not exist in this location, create a text file and save it to
that location.
2. Append a stanza for each regular expression as follows:

# transforms.conf

[<transform_stanza_name>]
REGEX = <regular expression>
FORMAT = <string>
WRITE_META = true
transform_stanza_name: A unique name for this stanza.
REGEX = <regular expression>: A regular expression that
defines how to match and extract metrics fields from this
metrics data.
FORMAT = <string>: A string that specifies the format of the
metrics event.
3. Create a data input for this source type as described in Set up a data input
for StatsD data, and select your custom source type.

For more about editing these configuration files, see About configuration files,
props.conf, and transforms.conf in the Admin Manual.

19
Example of configuring field extraction

This example shows how to create a custom source type and regular
expressions to extract fields from Graphite metrics data.

# props.conf.example

[graphite_plaintext]
TIME_PREFIX = \s(\d{0,10})$
TIME_FORMAT = %s
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
pulldown_type = 1
TRANSFORMS-graphite-host = graphite_host
TRANSFORMS-graphite-metricname = graphite_metric_name
TRANSFORMS-graphite-metricvalue = graphite_metric_value
category = Metrics

# transforms.conf.example

[graphite_host]
REGEX = ^(\S[^\.]+)
FORMAT = host::$1
DEST_KEY = MetaData:Host
[graphite_metric_name]
REGEX = \.(\S+)
FORMAT = metric_name::graphite.$1
WRITE_META = true
[graphite_metric_value]
REGEX = \w+\s+(\d+.?\d+)\s+
FORMAT = _value::$1
WRITE_META = true
Get metrics in from clients over HTTP or HTTPS

If you want to send metrics data in JSON format from a client that is not natively
supported to a metrics index over HTTP or HTTPS, use the HTTP Event
Collector (HEC) and the /collector REST API endpoint.

Create a data input and token for HEC

1. In Splunk Web, click Settings > Data Inputs.

2. Under Local Inputs, click HTTP Event Collector.
3. Verify that HEC is enabled.
1. Click Global Settings.
2. For All Tokens, click Enabled if this button is not already selected.
3. Click Save.

20
 4. Configure an HEC token for sending data by clicking New Token.
5. On the Select Source page, for Name, enter a token name, for example
"Metrics token".
6. Leave the other options blank or unselected.
7. Click Next.
8. On the Input Settings page, for Source type, click New.
9. In Source Type, type a name for your new source type.
10. For Source Type Category, select Metrics.
11. Optionally, in Source Type Description type a description.
12. Next to Default Index, select your metrics index, or click Create a new
index to create one.
If you choose to create an index, in the New Index dialog box:
1. Enter an Index Name.
2. For Index Data Type, click Metrics.
3. Configure additional index properties as needed.
4. Click Save.
13. Click Review, and then click Submit.
14. Copy the Token Value that is displayed. This HEC token is required for
sending data.

See Getting data in with HTTP Event Collector on the Splunk Developer Portal.

Send data to a metrics index over HTTP

Use the /collector REST API endpoint and your HEC token to send data
directly to a metrics index as follows:

http://<Splunk_host>:<HTTP_port>/services/collector \
-H "Authorization: Splunk <HEC_token>"
\
-d "<metrics_data>"
You need to provide the following values:

Splunk host machine (IP address, host name, or load balancer name)
HTTP port number
HEC token value
Metrics data, which requires an "event" field set to "metric".

For more about HEC, see Getting data in with HTTP Event Collector and Event
formatting on the Splunk Developer Portal.

21
For more about the /collector endpoint, see /collector in the REST API
Reference Manual.

Example of sending metrics using HEC

The following example shows a command that sends a metric measurement to a

metrics index, with the following values:

Splunk host machine: "localhost"

HTTP port number: "8088"
HEC token value: "b0221cd8-c4b4-465a-9a3c-273e3a75aa29"

curl -k https://localhost:8088/services/collector \
-H "Authorization: Splunk b0221cd8-c4b4-465a-9a3c-273e3a75aa29" \
-d '{"time":
1486683865.000,"event":"metric","source":"disk","host":"host_99","fields":{"region":"us-

22
Work with metrics

Search and monitor metrics

To analyze data in a metrics index, use mstats, which is a reporting command.
Using mstats you can apply metric aggregations to isolate and correlate
problems from different data sources. See mstats in the Search Reference
manual.

To enumerate metric names, dimensions, and values, use mcatalog, which is an

internal search command. See mcatalog in the Search Reference manual.

Other search commands do not work with a metrics index.

Note the following differences:

You cannot search metrics data for individual metric events.

You cannot perform search-time extractions.
You can enrich metrics with the equivalent of custom indexed fields, which
are treated as dimensions.
You can use reserved fields such as "source", "sourcetype", or "host" as
dimensions. However, when extracted dimension names are reserved
names, the name is prefixed with "extracted_" to avoid name collision. For
example, if a dimension name is "host", search for "extracted_host" to find
it.
Dimensions that start with underscore ( _ ) are not indexed, so they are
not searchable.

Search examples

To list counts of metric names over 10-second intervals:

| mstats count where metric_name=* span=10s BY metric_name

To perform a simple count of a dimension:

| mstats count where index=mymetricsdata

metric_name=aws.ec2.CPUUtilization

To calculate an average value of measurements for every 30-second interval:

23
| mstats avg(_value) WHERE index=mymetricdata AND
metric_name=aws.ec2.CPUUtilization span=30s

You can also display results in a chart. The following example uses a wildcard
search and group by:

| mstats avg(_value) prestats=t WHERE metric_name="cpu.*" span=1m by

metric_name | timechart avg(_value) as "Avg" span=1m by metric_name

This type of search can be used to stack different CPU metrics that add up to
100%.

This search shows an example of using an EVAL statement:

| mstats avg(_value) as "Avg" WHERE metric_name="memory.free.value"

span=5s | eval mem_gb = Avg / 1024 / 1024 / 1024 | timechart
max("mem_gb") span=5s

To list all metric names in all metrics indexes:

| mcatalog values(metric_name)

To list all dimensions in all metrics indexes:

| mcatalog values(_dims)

Use the REST API to list metrics data

You can also use the Metrics Catalog REST API endpoints to enumerate metrics
data:

Use the GET /services/catalog/metricstore/metrics endpoint to list

metric names.
Use the GET /services/catalog/metricstore/dimensions endpoint to list
dimension names.

24
 Use the GET
/services/catalog/metricstore/dimensions/{dimension-name}/values
endpoint to list values for given dimensions.

You can also use filters with these endpoints to limit results by index, dimension,
and dimension values.

See Metrics Catalog endpoint descriptions in the REST API Reference Manual.

Metrics indexing performance

This topic summarizes the results of metrics indexing performance.

Size on disk

When ingesting typical metrics payloads with supported metrics source

types (collectd_http, statsd, metrics_csv), a metrics index requires
about 50% less disk storage space compared to storing the same payload
in an events index.

Throughput

Consider the following when deciding whether to scale horizontally by

adding additional indexers.

Using the collectd_http source type with an HTTP Event Collector (HEC)
input, testing reached a constant of around 55,000 events per second
maximum ingestion throughput, and around 58,000 events per second
without additional search load.

The default batch size was 5,000 events per batch. A significant
difference in ingestion performance was not observed between
batch sizes of 100 to 5,000 events.
The keep-alive setting was enabled for these tests.
A typical event size was about 214 bytes.

Using the statsd source type with a UDP input, throughput was highly
variable depending on other network activity. For UDP inputs we
recommend using a universal forwarder as close as possible to where
metrics are collected.

Speed

25
 Consider the results from the following test for running metrics queries.
This test used metrics from 1,000 hosts, with a total event count of 6
billion events in the metrics index, where queries were representative and
did not use wildcards in metric_name.

Time range Events Query speed

1 hour 35 million < 0.1s
1 day 850 million ~3-5s
1 week 6 billion ~20-22s

See the Capacity Planning Manual.

Best practices for metrics

The following are best practices when working with metrics in the Splunk
platform:

For better performance, use the StatsD format with dimensions extension
(cpu.idle:0.5|g|#host:some-hostsplunk.com,app:some-app) rather than
the plain StatsD format that combines dimensions with the metric name
(cpu.idle.some-hostsplunk.com.some-app).

The _value field of a metric should be of type "Double", not type "String",
to avoid causing indexing inefficiencies.

For better search performance, enable hashing buckets by specifying

metric_name values in the splitByIndexKeys parameter in the
indexes.conf configuration file, then use one or more specific metric_name
values in the search WHERE clause. Tradeoffs are more index storage is
used, indexing throughput is slightly less, and searching with wildcards is
slower.

For a faster response time for REST calls, provide a time window when
calling the Metrics Catalog endpoint. By default, only the last 24 hours of
data is searched because this endpoint searches metadata, which does
not typically change often. See Metrics Catalog endpoint descriptions in
the REST API Reference Manual.

26
 Make sure dimension names do not start with underscore ( _ ), because
they will not be indexed.

27