Monitoring Linux and Windows Logs

with Graylog Collector

Bernd Ahlers
Graylog, Inc.

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Structured Logging & Introduction to
Graylog Collector

Bernd Ahlers
Graylog, Inc.

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Introduction: Graylog
Open source log management platform
Collect, index and analyze structured and
unstructured log data
Alerts based on log data
Extensible via custom plugins

Bernd Ahlers Graylog, Inc. bernd@graylog.com

Bernd Ahlers Graylog, Inc. bernd@graylog.com
Bernd Ahlers Graylog, Inc. bernd@graylog.com
Bernd Ahlers Graylog, Inc. bernd@graylog.com
Bernd Ahlers Graylog, Inc. bernd@graylog.com
Bernd Ahlers Graylog, Inc. bernd@graylog.com
Bernd Ahlers Graylog, Inc. bernd@graylog.com
Bernd Ahlers Graylog, Inc. bernd@graylog.com
Bernd Ahlers Graylog, Inc. bernd@graylog.com
 More about Graylog
www.graylog.org
marketplace.graylog.org
docs.graylog.org
github.com/Graylog2

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Why are we writing logs?
Getting insight & collecting business metrics
Debugging problems
Building an audit trail
Monitoring

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 How do we access our logs?
Applications write to local files
SSH into machines
tail, grep, awk
If lucky: central log management

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 What do they look like?
Syslog RFC 3164 (BSD)
Syslog RFC 5424

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Syslog RFC 3164 (BSD)

Nov 10 15:55:01 tumbler CRON[2684]: (root) CMD

(command -v debian-sa1 > /dev/null && debian-sa1
1 1)

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Syslog RFC 5424

2003-10-11T22:14:15.003Z mymachine.example.com
evntslog - ID47 [exampleSDID@32473 iut="3"
eventSource="Application" eventID="1011"] BOMAn
application event log entry...

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Apache

127.0.0.1 - bernd [28/Dec/2014:06:43:15 +0100]

"PROPFIND /remote.php/webdav/ HTTP/1.1" 207 910
"-" "Mozilla/5.0 (Linux) mirall/1.7.1"

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Postfix

Aug 5 17:05:26 hostname postfix/qmgr[308]:

A44F828C71: from=<bamm@example.com>, size=153136,
nrcpt=1 (queue active)

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Squid

sq18.wikimedia.org 1715898 2010-12-

01T21:57:22.331 0 1.2.3.4 TCP_MEM_HIT/200
13208 GET
http://en.wikipedia.org/wiki/Main_Page NONE/-
text/html - - Mozilla/4.0%20(compatible;%20MSIE
%206.0;%20Windows%20NT%205.1;%20.NET%20CLR
%201.1.4322) en-US -

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 log4j

0 [main] INFO MyApp - Entering application.

36 [main] DEBUG com.foo.Bar - Did it again!
51 [main] INFO MyApp - Exiting application.

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Ruby Logger

I, [2015-11-18T00:16:27.723972 #3609] INFO -- :

Hello world!

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 #1 Problem: Timestamps
Everyone likes to invent one
Missing most of the time: timezone, year

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 How to get value out of unstructured logs?

Regex
More regex
Even more regex

Bernd Ahlers Graylog, Inc. bernd@graylog.com

((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:
[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4})
{1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-
9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:
[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-
4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]
{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-
9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]
{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-
5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d))
{3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-
Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]
{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|
1\d\d|[1-9]?\d)){3}))|:)))(%.+)?

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Grok
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9...

USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-
Za-z-]{0,62}))*(\.?|\b)
EMAILLOCALPART [a-zA-Z][a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILLOCALPART}@%{HOSTNAME}
...
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Graylog: Extractors
Regular expressions based
Extracts data into message fields

Bernd Ahlers Graylog, Inc. bernd@graylog.com

Bernd Ahlers Graylog, Inc. bernd@graylog.com
 How to fix this?
Central log collection (Graylog, ELK, others)
Use structured log formats
Structured Syslog RFC 5424
CEF Format
GELF
JSON

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Structured Syslog RFC 5424
2003-10-11T22:14:15.003Z mymachine.example.com
evntslog - ID47 [exampleSDID@32473 iut="3"
eventSource="Application" eventID="1011"] BOMAn
application event log entry...

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 CEF by ArcSight/HP
Sep 19 08:26:10 host CEF:0|HP|siem|
1.0|100|service
successfully stopped|10|
src=10.0.0.1 dst=2.1.2.2 spt=1232

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 GELF
{ "version": "1.1",
"timestamp": 1385053862.3072,
"host": "example.org",
"short_message": "A short message",
"full_message": "Backtrace here\n\nmore stuff",
"level": 1,
"_user_id": 9001,
"_some_info": "foo",
"_some_env_var": "bar"}

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 JSON
{ "source": "example.org",
"message": "A log message",
"timestamp": "2015-11-15T10:43:21Z",
"user_id": 9001,
"http_method": "GET"}

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 How we try to improve the ecosystem
Icinga2 GELF output for events
Docker GELF logging driver (since Docker 1.8)
apache-mod_log_gelf (beta)
log4j2-gelf
gelfclient Java library
svloggelfd (log forwarding for runit)

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 We at Graylog <3 structured data
and you should too!

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Introduction: Graylog Collector
Reads local log files and ships them to Graylog
Windows EventLog support (limited for now)
Transport encryption via TLS
Runs on Linux, Windows, Mac OS X and AIX

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Why another Collector?
There are lots of others: nxlog, fluentd, heka,
filebeat, rsyslog, syslog-ng
We want integration and centralized
management of collectors in Graylog

Bernd Ahlers Graylog, Inc. bernd@graylog.com

Bernd Ahlers Graylog, Inc. bernd@graylog.com
 Collector Installation
OS packages for Linux distributions
Manual installation on Windows via ZIP file
(MSI upcoming)
Runs as Windows service

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Collector Configuration
server-url = "http://your-graylog-server:12900"
inputs {
windows-application-log {
type = "windows-eventlog"
source-name = "Application"
}
}
outputs {
gelf-tcp {
type = "gelf"
host = "your-graylog-server"
port = 12201
}
}

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Collector: Current State
Windows EventLog support needs update to
support new Windows APIs
File reading needs improvement
Centralized management needs to be
implemented
:-(

Bernd Ahlers Graylog, Inc. bernd@graylog.com

 Tomorrow: Hackathon

Bernd Ahlers Graylog, Inc. bernd@graylog.com

Thank you!

Thank you for your time!

Bernd Ahlers Graylog, Inc. bernd@graylog.com

QA

Ask me anything!

Bernd Ahlers / Graylog, Inc.

bernd@graylog.com
@berndahlers
www.graylog.org
github.com/Graylog2

Bernd Ahlers Graylog, Inc. bernd@graylog.com