LABSEC-3336 – Network

Forensics & Threat Awareness

Stefano De Crescenzo – S&RO
Panos Kampanakis – S&TO
Joseph Karpenko – S&RO
Agenda
• Introduction
• Part 1 – On-Device Detection and Forensic
• Part 2 – Detection and Forensic using Telemetry Data
• Best Practice and Hardening

3
The Challenges Come from Every Direction

Sophisticated Complicit
Attackers Users

Dynamic Boardroom
Threats Engagement

Defenders

Complex Misaligned
Geopolitics Policies

4
5
DoS vs APT on Infra Device – Threats Landscape

6
Compromised Network Infrastructure Devices and
Endpoints Differences
• Infrastructure devices are a central point for information exfiltration
• Many features exist to identify and exfiltrate interesting network traffic
• Forensic and analysis tools not as developed as with endpoints
• Device OS protections and hardening might not be as mature as with
endpoints
• End-to-end encryption could alleviate part of the concern. Man-in-the-
middle could still pose a threat

7
Attacker Motivations

• Traffic Exfiltration
• Traffic Re-routing
• Access to Crypto Material
• NAT-ing
• DoS
• Etc…

8
Potential Attack Methods

• Commands
• Manipulating Cisco IOS Images
• Vulnerabilities
• Attack Vectors
• Alter the booted IOS image
• Tamper the run-time memory
• Access the Linux Shell (XE only)
• Modify the ROM Monitor
• HW attacks
• Combination of these

13
 Possible Attack Methods
Attack Vector Privileges Required Recommended Best Practices

Use Authentication, Authorization, and Accounting

Code injection in run time via IOS commands Admin Use TACACS+ Authorization to Restrict Commands

Implement Credentials Management

Implement Configuration Controls
Cisco Secure Boot
Digitally Signed Cisco IOS Images
Maintain Cisco IOS Image File Integrity
Modified Binary Image Admin Implement Change Control
Harden the Software Distribution Server
Implement Credentials Management
Protect Interactive Access to Devices
Cisco Secure Boot
Cisco Image Signing
Implement Change Control
Modified ROMMON Image Admin
Harden the Software Distribution Server
Implement Credentials Management
Protect Interactive Access to Devices
Hardware Modification Physical Cisco Supply Chain Security

Vulnerabilities that could cause writing in memory Depends Keep Cisco IOS Software Updated

14
Malwares found on Customer Networks

15
Anything in Common?
• No Vulnerability Exploitation
• Attackers gets in and perform the changes by using admin
credentials or physical access!
21
How difficult it is to create a malware for IOS?
• One of the first research demonstrating
binary modification of IOS images was
published in 2004 (IOS Shellcode by
Mike Lynn)
• Many others in the following years (not
complete list!)
• Killing the myth of Cisco IOS – 2008
• Cisco IOS Router Exploitation – 2009
• Killing the Myth of Cisco IOS Diversity
– 2011
• Writing Cisco IOS Rootkits - 2015

22
Common Issues

• Default or Easy Credentials • No infrastructure access list

• Old Firmware Images • No AAA
• Outdated Harware (in some • Templates
cases EoL)
• Many mores
• Telnet enabled (yes in 2015!)
• SNMP read/write with default
community strings

23
Lab Overview

24
 Lab Environment for Students

Student
Lab Internet

Internet
PodX
(with 2
Infrastructure
R1-ASA5510 Sw1 RTRs) Sw2 Rtr1

Terminal Servers 1 and 2

SiLK ELK

26
IOS Per-Pod Network
10.0.0.0/8 End-hosts: 172.16.<pod#>.0/28
192.168.0.0/16 Infrastructure: 172.16.<pod#>.60/30
Infrastructure: 172.16.<pod#>.64/29
.1
10.<pod#>.11.0/24 .1 10.<pod#>.21.0/24 Infrastructure
.2

SW podX-rtr1 .2

10.0.0.0/8 .2
SW 192.168.0.0/16
10.<pod#>.22.0/24
.2 10.<pod#>.12.0/24 .1 .1 SiLK: 10.0.1.32
ELK SiLK ELK: 10.0.1.33
podX-rtr2
NetFlow export

Flow Collection / Analysis

27
 Reference
Logging into the lab Reference
To VPN into the lab Slide
Slide
• Open the ( ) Cisco AnyConnect
VPN Client from the Programs
menu
• “Connect to” 64.102.242.66
• Use credentials
• Username: [see proctor]
• Password: [see proctor]
• Accept the Certificate warnings that
may appear

28
Materials to take with you
• Go to https://cisco.app.box.com/files/0/f/4929794837/
• The directory contains
• /captures of suspicious packets/ – PCAP captures of the C&C and exfiltration packets in
the lab
• /initial configs/ – Initial, base configurations for all routers
• /solution videos/ – Videos of the solutions of the labs
• Presentation PDFs – The presentation, lab guide and solutions.

31
Part 1 – On Device
Detection

32
On Device Detection

• Verify integrity of the IOS image

• Verify Integrity of Run-Time memory
• Verify Integrity of ROMMON
• Watch for on-device Indicators of Compromise

33
Verify IOS Image Integrity

34
Verifying the Integrity of IOS Image

Several Techniques
• Use the verify md5 command
verify /md5 filesystem:filename [md5-hash]
• Use the image verification feature
file verify auto
• Use Offline Image File Hashes
http://www.cisco.com/c/dam/assets/about/security/resources/ioshashes.zip
(updated every week)
Note: All these methods will not verify the run-time memory!

35
 Verify Authenticity of Digitally Signed Images

• IOS and IOS-XE support digitally signed images on some platforms

• Use the show software authenticity file command to verify the
authenticity of an image on the flash
• Use the show software authenticity running (IOS only) to verify the
authenticity of the image running currently
Router# show software authenticity file c1900-universalk9- Router#show software authenticity running
mz.SPA.152-4.M2 SYSTEM IMAGE
File Name : c1900-universalk9-mz.SPA.152-4.M2 ------------
Image type : Production Image type : Production
Signer Information Signer Information
Common Name : CiscoSystems Common Name : CiscoSystems
Organization Unit : C1900 Organization Unit : C1900
Organization Name : CiscoSystems Organization Name : CiscoSystems
Certificate Serial Number : 509AC949 Certificate Serial Number : 509AC949
Hash Algorithm : SHA512 Hash Algorithm : SHA512
Signature Algorithm : 2048-bit RSA Signature Algorithm : 2048-bit RSA
Key Version : A Key Version : A Verifier

Information Verifier Name : ROMMON 1

LAB – Getting Connected and Task1

• Open the Lab Guide and connect to Router 1 or Router 2 in your pod
• Please note that Pod1 to Pod10 have a 2800 while Pod11 to Pod15
have a 2900. Watch out for specific instructions in the lab guide.
• Task 1 – Verify the integrity of the image on the flash

37
Verify the Integrity of Run-Time
Memory

38
Verifying the Integrity of the IOS or IOSd run-time
memory
• There is no way to reliably verify all parts of the run-time memory
• IOS-XE adds additional complexity as the attacker may compromise the
Linux part of the system or other additional subsystems such as the QFP or
linecards
• The methods proposed are based on commands and output generated by
the IOS or IOSd. An attacker could modify the output of the commands to
match what is expected and invalidate the methods presented
• It is very important to follow security best practices included in the IOS
Hardening guide and make sure that the administrative credentials are
protected

39
Verify the IOS (IOSd) memory – Text Region
Virtual Memory
• When loaded in memory, Cisco IOS or IOSd would
expand in several memory regions. One of them is Main region
the text region and contains the executable code.
Text Region
• Because the text region contains the Cisco IOS or
IOSd code, this region should not change during Data Region
reboot and should be the same across similar
devices (i.e. same hardware running the same BSS Region
software release and feature set).
• Cisco provides ASLR for some Cisco IOS or IOS-XE Heap Region
SW releases. When an image boots with ASLR
enabled, the code in the text region will be modified
to take the offsets into consideration. The method
presented here cannot be used in that case. IOMEM Region
IOS-XE Architectural Differences
TCAM4
Resourc Packet
Virtual Memory e DRAM Buffer SRAM

IOSd text Region

Processor pool
IOSd data Region PPE0
PPE0
PPE0
PPE0 QFP
PPE0 PPE0
PPE2 PPE0
PPE0 PPE0
PPE4
PPE0
PPE0 PPE0
PPE3
PPE1
Libraries
PPE0
…
PPE0 PPE0
…
PPE0
PPE0 PPE0 PPE0
Buffer,Buffer, queue, (BQS)
PPE0 PPE6 PPE0
PPE5 PPE40 queue, schedule
schedule (BQS)

Dispatcher/
Heap Region Packet
Buffer

IOSd IOSd SPA

QFP drivers
active standby

RP ESP SIP
Platform adaptation layer Platform adaptation layer Platform adaptation layer
Linux kernel Linux kernel Linux kernel
ASLR Implication

• Depending on the HW and SW combination, Address Space Layout

Randomization (ASLR) may be enabled on this system
• On IOS, ASLR is implemented directly into the IOS code
• On IOS-XE, ASLR is provided by Linux
• On IOS, two types of ASLR exist (and can coexist):
• RTO : Move the text section by a random offset (usually the last 4 digits of the text starting
address)
• RDO : Move data and bss. This means that the pointing addresses are changed within the text
section, so the content of the text section changes between two system or on the same system
when reloaded.

42
 ASLR Implementation Types

Link-time
text data

Run-time RTO
text RDO
data

0x0 Offset value 0xFFFFFFFF

43
How to verify if ASLR is enabled?

• Compare the show region output with another identical

system (same HW and SW)
router#show region router#show region
Region Manager: Region Manager:
Start End Size(b) Class Media Name Start End Size(b) Class Media Name
0x16000000 0x17FFFFFF 33554432 Iomem R/W iomem:(iomem) 0x16000000 0x17FFFFFF 33554432 Iomem R/W iomem:(iomem)
0x60000000 0x75FFFFFF 369098752 Local R/W main 0x60000000 0x75FFFFFF 369098752 Local R/W main
0x6001B5B8 0x6487FFFF 75909704 IText R/O main:text 0x6001EDF8 0x6487FFFF 75895304 IText R/O main:text
0x6488BC40 0x6692125F 34166304 IData R/W main:data 0x6488F480 0x66924A9F 34166304 IData R/W main:data
0x66921260 0x6742621F 11554752 IBss R/W main:bss 0x66924AA0 0x67429A5F 11554752 IBss R/W main:bss
0x67426220 0x75FFFFFF 247307744 Local R/W main:heap 0x67429A60 0x75FFFFFF 247293344 Local R/W main:heap
0x80000000 0x95FFFFFF 369098752 Local R/W main:(main_k0) 0x80000000 0x95FFFFFF 369098752 Local R/W main:(main_k0)
0xA0000000 0xB5FFFFFF 369098752 Local R/W main:(main_k1) 0xA0000000 0xB5FFFFFF 369098752 Local R/W main:(main_k1)
0xF6000000 0xF7FFFFFF 33554432 Iomem R/W iomem 0xF6000000 0xF7FFFFFF 33554432 Iomem R/W iomem

44
Verification Flowchart

Most likely the

Extract main:text Compute the hash text section was
not modified

Router Under Evaluation Is the Yes

hash
the
No
same?

Extract main:text Compute the hash Contact Cisco for

further analysis

Router Known-not-to-be-compromised

45
How to extract main:text ?

• 2 Methods
• By creating a coredump
• By using the system:memory file system (this method cannot be used on
3900E, CGR1000, Catalyst 6880, Cisco 1800 and Cisco 4945)
• Drawbacks! Any?

47
Method 1 : Create a coredump

• Cisco IOS is able to produce a full memory dump (= coredump) which

includes a copy of the text region run in memory.
• This is configured via the exception command and can be generated via
the write core command
• dd or any other similar utility can be used to extract the text region once the
core dump is available

48
 Method 1 - Example

1. Configure the exception and exporting method (FTP)

exception core-file <name> compress

exception protocol ftp exception region-size 65536
exception dump <ip address of the ftp server>
ip ftp username <user>
ip ftp password <pass>

2. Create a coredump via the write core command

49
Method 1 – Example (2)

3. Calculate the boundary and size for text memory extraction

router#show region
Region Manager:
Start End Size(b) Class Media Name
0x08000000 0x0BFFFFFF 67108864 Iomem R/W iomem
0x40000000 0x4BFFFFFF 201326592 Local R/W main
0x401012B8 0x44157FFF 67464520 IText R/O main:text
<output truncated>

4. Use dd to extract the text region (skip is 0x1012B8 in decimal)

dd if=<corefile> bs=1 count=67464520 skip=1053368 of=router_main_text

5. Calculate the MD5 of the text region and compare with the MD5 of
a good known text region

50
Method 2 : Using the system:memory file system

Alternatively to create a core dump, the text region can be exported directly via copy
command
1. Locate the text region via the dir command
router#dir system:/memory
Directory of system:/memory/
<output truncated>
10 -r-- 268402688 <no date> more_heap
6 -r-- 67464520 <no date> text

2. Export the text region via the copy command

router#copy system:memory/text ftp:

3. Calculate the MD5 of the text region and compare with the MD5 of a good known
text region

51
Computing the hash on the router itself

• You could also use the verify /md5 command to compute the hash of the
text region directly on the router :
router#verify /md5 system:memory/text
[...]
Done! verify /md5 (system:memory/text) = 1edd0985da7f1a490729fd0aaf9c0bd7

• This value should be compared with the MD5 of a good known text region
NOTE: The MD5 value obtained with any of the three methods presented
should be the same

52
 LAB – Task 2 and 3
• Task 2 – Verify whether ASLR is enabled.
• Task 3 – Verify the integrity of the run-time memory
• 3.1 – Use the verify /md5 command to calculate the MD5 hash
• 3.2 – Generate the coredump and verify that the MD5 of the text section
match the one calculated in task 3.1
• 3.3 – Export the text section via the copy command and verify that it matches
the value in task 3.1

55
Verify the ROMMON Integrity

56
Verify ROMMON (ROM Monitor) Integrity

• Bootloader of Cisco routers/switches (RP / SP / modules)

• Upgradeable via the CLI
• Version visible with show version (IOS) or show platform (IOS-XE)
• Located at a fixed address in memory, which is platform dependent
• C2900 and C7600/RSP720 provide memory address information in the output
of show region command.
• Other platforms do not provide the information about ROMMON memory
addresses via IOS CLI. We need to inspect the code to see where the
ROMMON code is loaded.
• For IOS-XE is required to have shell access to be able to dump the ROMMON
memory

57
ROMMON in Memory (Image)
Non Upgraded ROMMON Upgraded ROMMON

Boot -> 0xbfc00000

RM1

boot -> 0xbfcc0000

RM2

Virtual Memory 58
CLI commands to dump ROMMON(IOS only)

• IOS
• service internal
• show memory 0x<start> 0x<end>

• ROMMON
• priv (may require a password)
• dump <address> <length>

59
Additional Indicators of
Compromise

60
Additional Indicators of Compromise
gdb *
• Some commands can be used and abused to modify the test *
tlcsh *
behavior of Cisco IOS and during reconnaissance attacks debug *
service internal
• Administrator should check all available logging facilities config-register *
for the presence of such commands and investigate boot *
upgrade *
further attach *
hw-module * (XE)
• Check within the core dump (e.g. via string utility) remote *
• Check the command history via show history all ipc-con *
if-con *
command execute-on *
• Check command accounting logs service-monitor *
platform shell (XE)
• Check syslog for unusual connection request *
show region
• Check booting information via show version command show memory *
• Check ROM Monitor information via set command show platform *
do-exec

61
Analyze the command history

• Analyzing the command history could add information to the

investigation process. The attacker, in fact, may have left behind
some traces of commands that he executed
• Use the show history all command or extract the commands from
the coredump
• Look for the presence of any of the commands listed in the unusual
and suspicious commands section
• Look for the presence of commands that should not be there or
dummy commands used to fill in the history buffer
63
 Analyze the command history - Example
• In this example we use strings to search for all commands in the
coredump
$ strings <CORE> |grep ^CMD:
CMD: 'verify /md5 system:memory/text' 06:59:50 UTC Wed Jan 15 2014
CMD: 'service internal | i exce' 07:02:41 UTC Wed Jan 15 2014
CMD: 'conf t' 07:02:45 UTC Wed Jan 15 2014
CMD: 'exception flash procmem bootflash:' 07:02:54 UTC Wed Jan 15 2014
CMD: 'exception core-file CORE compress ' 07:03:31 UTC Wed Jan 15 2014

64
Useful information in “show version”

• Use show version to check

• Date of the last reload
• Reason for the last reload
• Verify that the booted file is pointing to the correct image
• Verify that the config-register has not been modified (0x2102)

65
ROMMON variables

• Check ROMMON and verify that: (note this will require a reload)
• BOOT variable is correct
• OFFSET/NO_CPU/NO_RANDOM_NUM variables are not set (this can be
used to reset ASLR)
• GDB variable is not set (can be used to re-enable GDB on image where this is
disabled)
• For IOS-XE only, the following variables are not set
• REAL_MGMTE_DEV
• DEBUG_CONF
• ROMMON_DEBUG_CONF
• BOOT_PARAM
• SR_NVRAM_PATH
• SR_MGMT_VRF
• STBY_CONS_EN

66
ROMMON Information

• Use the show rom-monitor to verify whether the ROMMON has been
upgraded
• The output may change depending on the platform
• Some device (example 7600 and 6500) have two rommon files (RM1
and RM2) and two upgradable regions (F1 and F2)
• Remember to check the ROMMON for all RPs and SPs
• Make sure to note any unusual output. Examples:
• ROMMON has been upgraded but this was not a scheduled changes
• The initial ROMMON and the upgraded ROMMON show the same ROMMON
version
67
Example of ROMMON Info
1841#show rom-monitor 7603-3#show rom-monitor slot 1 rp
ReadOnly ROMMON version: Region F1: INVALID
Region F2: INVALID
System Bootstrap, Version Currently running ROMMON from S
12.4(13r)T, RELEASE SOFTWARE (fc1) (Gold) region
Technical Support:
http://www.cisco.com/techsupport
Copyright (c) 2006 by cisco
Systems, Inc.

No upgrade ROMMON programmed or not

yet run
Currently running ROMMON from
ReadOnly region
ROMMON from ReadOnly region is
selected for next boot

68
Analyze the file system content
• Some attacks (for example TCL attacks) may require the attacker to store file on the file
system
• Good practice is to regularly review the content of the file system and investigate any
anomaly
• dir all-filesystems command can help visualizing content, timestamps and size of a file
• Be aware that it includes more than just flash. Use show file system to visualize all
filesystem directories
router#dir all-filestystems
[…]
Directory of disk0:/
1 -rw- 133283588 Dec 31 2013 08:16:18 +00:00 c7600s72033-advipservicesk9-mz.122-33.SRD4
2 -rw- 3002 Dec 21 2011 09:56:56 +00:00 ciscoapi.tcl

69
Check the TLB (IOS only)
The Translation Lookaside Buffer (TLB) is a mechanism to map virtual memory regions to
physical memory regions.
Entries in the TLB have permissions : R (Read), W (Write), X (eXecute)
A crash occurs if the permissions are not respected
Rule (NX, DEP, X-space, X^W) : a region should never be writeable and executable.
• Text section should be executable only
• Data/Heap/… sections should be writeable only
Commands are platform specific:
• show platform
• show platform tlb
• show platform hardware tlb

70
Check the TLB – Continued (IOS only)
router#show region
• To perform the TLB verification, use Region Manager:

the show region command to Start

0x08000000
End
0x3BFFFFFF
Size(b)
872415232
Class
Local
Media
R/W
Name
main
identify the text section boundaries 0x08000000 0x0FFFFFFF 134217728 IText R/O main:text
0x10000000 0x111E89F7 18778616 IData R/W main:data
0x111E89F8 0x12B0A7D3 26353116 IBss R/W main:bss
• Use the show platform tlb (or show 0x12B0A7D4 0x3BFFFFFF 693065772 Local R/W main:heap
0x3C000000 0x3FFFFFFF 67108864 Iomem R/W iomem
platform) command to see the
information about TLB entries. Verify router#show platform tlb
that the memory where the text Variables Size TLB's......
----------------------
section is stored is still read-only Virt Address range Phy Address range W-I-M-G Attr ESEL
=====================================================================
0xFFC00000-0xFFFFFFFF 0x0_FFC00000-0x0_FFFFFFFF 0-0-1-1 R-X (0)
0xFEC00000-0xFEFFFFFF 0x0_FEC00000-0x0_FEFFFFFF 0-1-1-1 RWX (1)
0xFFB00000-0xFFBFFFFF 0x0_FFB00000-0x0_FFBFFFFF 0-1-1-1 RWX (3)
0x04000000-0x07FFFFFF 0x0_04000000-0x0_07FFFFFF 0-0-1-0 RWX (6)
0x08000000-0x0BFFFFFF 0x0_08000000-0x0_0BFFFFFF 0-0-1-0 R-X (7)
0x0C000000-0x0FFFFFFF 0x0_0C000000-0x0_0FFFFFFF 0-0-1-0 R-X (8)
0x10000000-0x1FFFFFFF 0x0_10000000-0x0_1FFFFFFF 0-0-1-0 RWX (9)
0x20000000-0x2FFFFFFF 0x0_20000000-0x0_2FFFFFFF 0-0-1-0 RWX (10)
0x30000000-0x3FFFFFFF 0x0_30000000-0x0_3FFFFFFF 0-0-1-0 RWX (11)
[…]

71
Check Platform Shell Logs (IOS-XE only)

• IOS-XE allows the device administrator to access the Linux OS as root.

• This poses a big security risk in case the admin credentials are compromised
• 3 types of images :
• No restriction (ASR1002, ASR1004, ASR1006, ASR1013)
• Shell license required (Cisco only) (ASR1001, ASR1001-X, ASR1002-X, ISR-4xxx, CSR-1000v)
• Challenge/response mechanism (C3850, C4500x, ..)
• Whenever the linux root shell is accessed, a log file is created on the file system which
logs any command the user executes
• It is important to look for the presence of such files and review the content to detect
any unauthorized access
• Whenever the platform shell is accessed, a syslog is also issued to inform about this
event

72
 IOS-XE Shell access monitoring
*Jun 18 19:06:34.630: %IOSXE-5-PLATFORM: SIP0: %SYSTEM-3-SYSTEM_SHELL_LOG: Shell ended: con 0
*Jun 18 19:06:34.670: %IOSXE-5-PLATFORM: SIP0: %SYSTEM-3-SYSTEM_SHELL_LOG: Log: harddisk:tracelogs/system_shell_R0.log.20140618190625
*Jun 18 19:06:34.678: %IOSXE-5-PLATFORM: SIP0: %SYSTEM-3-SYSTEM_SHELL_LOG: (fingerprint: 6346d70a3e4928fbb8375e85e8c8983c)

Router#cd tracelogs
Router#dir | include system_shell
843706 -rw- 529 Jun 18 2014 19:19:01 +00:00 system_shell_R0.log.20140618191728
843699 -rw- 161 Jun 18 2014 19:06:34 +00:00 system_shell_R0.log.20140618190625

Router#more tracelogs/system_shell_R0.log.20141127102349
Script started on Thu Nov 27 10:23:50 2014
[Router:/]$ ps
PID TTY TIME CMD
24132 pts/1 00:00:00 bash
24165 pts/1 00:00:00 ps

73
LAB – Task 4 and 5

• Task 4 – Verify the TLB permission

• Task 5 – Verify the presence of suspicious commands in the
coredump

74
Coredump Forensic and Capture
The Flag challenge

80
Verification Flowchart

Extract main:text Compute the hash

Router Under Evaluation Is the

hash
the
No
same?

Extract main:text Compute the hash Contact Cisco for

further analysis

Router Known-not-to-be-compromised

81
Coredump Differential Analysis

• Use radiff2 or other tools to find out the difference between a known-
good text region and the compromised one
• For each offset resolve the original memory address
• Decode the memory addresses to find out the function names that
have been modified
• Browse the source code to understand what those functions are doing
• This will tell which function was modified but not what the injected code
is doing (this will require to reverse engineer the injected code to
understand what it is doing)

82
Task 7 - Capture The Flag – BONUS LAB

• Task 8 – You have received a coredump from a router that is

suspected to be compromised
• 8.1 – Determine whether this router was compromised by checking the
integrity of the text section
• 8.2 – Determine where the text section differs from a know-not-to-be-
compromised text section
• 8.3 – Try to understand which function was compromised

The first to say which function was modified and can demonstrate
which vulnerability was injected wins a PSIRT Polo Shirt!

83
 Part 2 – Detection and
Forensic using Telemetry
Data

84
Telemetry Collection

85
Attacker’s Methodology
• Overt Channels • Covert Channels
• FTP • Encrypted
• HTTP • Tunneling
• IM • Protocol Hopping
• Email • Steganography (i.e. over voice)
• Telnet • Timing channels
• SIP • Deliberately malformed packets
• ICMP (HICCUPS system)

86
This lab goals for this part of the lab
• Describe possible network device compromises
• Command and Control (C&C) communication indicators of compromise (IOCs)
• Data exfiltration indicators of a compromise (IOCs)

• Discuss
• Potential compromise vectors, NOT ALL of them
• Tools that could be used to identify these IOCs
• Policy and instrumentation best practices that can identify and protect against these
IOCs
• Questions?

87
Cisco IOS NetFlow

88
Network Telemetry - NetFlow
• NetFlow is telemetry pushed from routers/switches
• Each device can be a sensor
• Simple summary of connections
• Negligible performance impact on routers

• Not just Cisco

• Like a phone bill
• Packet capture is like a wiretap
• NetFlow data can be collected and relayed to
multiple tools

89
What Is a Traditional IP Flow?
1. Inspect a packet’s seven key fields and identify the values
2. If the set of key field values is unique, create a flow record or cache entry
3. When the flow terminates, export the flow to the collector

1
NetFlow
NetFlow Export
Key Packets
Fields

2
Reporting
3
NetFlow
Internal Threat Information Resource
• NetFlow is available on routers and switches
• Have syslog-like information without having to buy a firewall
• One NetFlow packet has information about multiple flows

Header
Flow Flow
• Sequence number
• Record count
Record
… Record
• Version number

NetFlow Cache Export Packets

• Approximately 1500 bytes
• Typically contain 20–50 flow records
• Sent more frequently if traffic increases on NetFlow-enabled interfaces
Netflow v5 configuration Netflow collector and
port

ip flow-export version 5
ip flow-export destination 10.0.1.32 9995
interface FastEthernet0/0
Interface to collect
ip flow ingress data from

93
Telemetry Analysis

125
Network Telemetry Analysis tools
• SIEM tools can consume network telemetry (logs, NetFlow, captures) and
provide insightful information, search and reporting capabilities.
• In the interest of time, in this lab we will use open-source ELK.
• Reference slides include information about other tools
• Lancope
• SiLK

126
ELK: Elasticsearch,
Logstash, Kibana

127
Overview of ELK
• Elasticsearch
• Indexes data
• Allows searching and analyzing in near real time

• Logstash
• Reads data from multiple inputs
• Provides filtering for scrubbing, parsing, and enriching data
• Outputs data in multiple formats

• Kibana
• Visualization & Exploration
• Interactive dashboards can be created with many panels

• All open source

128
Elasticsearch
• Indexes structured and unstructured data
• Every field is indexed and searchable
• Distributed, clustered architecture allows for scaling
• RESTful API to interact with data
• String fields support Regex and ranges (using TO operator)
• Guide
• http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/index.html
• Reference
• http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index.html

129
Logstash
• Receives data through inputs
• Processes data using codecs
• Manipulates data with filters
• Sends parsed data via outputs
https://www.elastic.co/guide/en/logstash/current/introduction.html

130
Kibana
• Graphical search and analytics dashboard for Elasticsearch
• Base consists of a Query Row and a Filter Row

• Additional rows can be added

• Panels can be added to each row
• List of available panels in documentation
• http://www.elasticsearch.org/guide/en/kibana/current/index.html
133
ELK Queries vs Filters
• Queries
• Similar to a filter, but also asks “How well does this document match?”
• When queries are used, documents are assigned a relevance _score
• Used for sorting matching documents by relevance
• Note for the Lab: Panels need to use the right Query source. Use the panel edit icon to set the Query
tab appropriately. Then the panel will refresh and only reflect the data matching the query

• Filters
• Used to ask every document a yes/no question
• For fields that contain exact values
• Caches results
• There is a performance advantage for using filters over queries
• They can be used in conjunction.
• Best practice is to use queries on a set of documents that has already been filtered
134
Query Bar

135
Kibana – Lab Dashboard panels

137
NetFlow analysis with ELK
• Pie-chart panels

138
 ELK Traffic Analysis Example
• Query both src/dst for 172.16.1.0 for the last 24 hours
• Filter or Query used ‘ipv4_src_addr_txt:172.16.1.* OR
ipv4_dst_addr_txt:172.16.1.*’

140
NetFlow ICMP records
• NetFlow uses the destination port field in ICMP flows to encode the Type and Code
• The ICMP Type and Code are concatenated together and then encoded as decimal
• Type 3, Code 3 (ICMP Port Unreachable) = 0303 (hexadecimal) => 0771 (decimal)

• Convert the decimal value to hexadecimal, some common examples follow

• DECIMAL HEXADECIMAL ICMP MESSAGE
• 0000 0000 Echo Reply
• 0768 0300 Network Unreachable
• 0769 0301 Host Unreachable
• 0771 0303 Port Unreachable
• 0772 0304 Fragmentation needed but DF bit set
• 0773 0305 Source Route Failed
• 0777 0309 Communication with Destination Network Admin Prohibited
• 0781 0313 Communication Administratively Prohibited
• 2048 0800 Echo Request
• 2816 1100 TTL Exceeded in Transit

141
C&C communications

173
C&C Topologies
• Centralized
• Fast command dissemination
• Easier to detect by the victim

• Peer-to-Peer
• Decentralized
• Harder for the victim to detect
• Complex to setup

174
C&C Methods
• Attackers use well known services to hide C&C communications
• Some network protocols are especially suited for tunneling traffic into the
network
• C&C candidate protocols
• ICMP Tunneling
• HTTP/S
• DNS Tunneling

175
 Potential C&C Communication
ICMP

Internet

C&C

Unsolicited ICMP
messages

Potential ICMP types include:

• Echo Reply
• Destination Unreachable
• Redirect
• Time Exceeded
176
Potential C&C Communication: DNS
DNS
• DNS is a critical internet service
• RFC 1034 and 1035
• Multiple types of resource records (RR)
• UDP port 53 and TCP port 53
• TCP is used for DNS Zone transfers and legacy DNS messages greater then
512 bytes
• DNS Best Practices, Network Protections, and Attack Identification
http://www.cisco.com/web/about/security/intelligence/dns-bcp.htm

179
Potential C&C Communication
DNS Tunneling
DNS transaction can tunnel
C&C communications between
the victim and its master
C&C

Internet

180
 Potential C&C Communication:
HTTP
HTTP sourced from the router
pulling C&C commands from
their master using HTTP
C&C

Internet
Internet

Detection:
Should your network devices use HTTP outbound?
183
Data Exfiltration

185
Data Exfiltration communication
Packets Egressing without Ingressing
• Compromised router is examining transit traffic and generating new packets
containing the captured data
• Key indicator: The router is creating these new packets, the packets are not
associated with ingress transit traffic

Packets egressing
No bidirectional flows

Internet

Packets egressing
after ingressing

186
Data Exfiltration communication
Traffic Spikes
• Compromised router is examining transit traffic and generating new packets
containing the captured data
• Key indicator: Traffic spikes above baseline
• Possibly to specific destinations

Exfiltrated data egressing

to specific destinations

Internet

194
Data Exfiltration communication
Unidirectional connectionless protocols
• Compromised router is examining transit traffic and generating new packets
containing the captured data
• Key indicator: Unidirectional connectionless protocols (for example UDP or
ICMP)

Exfiltrated data egress using

unidirectional connectionless protocols

Internet

198
Data Exfiltration communication
Tunneling
• Many traffic tunneling options exist
• Basic network encapsulation
• GRE
• Teredo and other IPv6 tunneling options
• IPSec

• Application tunneling
• ICMP Tunneling
• DNS tunneling – data can be passed thru DNS lookups. High volumes of requests to
suspicious domains can be indicators

203
Investigating suspicious traffic further
• Not all traffic that matches an indicator is bad
• Additional investigation is required to confirm the IoC
• Packet captures should be used to record potential IOCs for further investigation
• Packet capture protocol analysis and payload inspection can help validate
malicious traffic
• Payloads may be obfuscated to make detection more difficult
• Encryption
• XOR
• Many other methods

204
Network integrity lab
Lab Internet
Potential C&C traffic to
compromised infrastructure
router/s in
172.16.<pod#>.60/30 or End-hosts: 172.16.<pod#>.0/28
172.16.<pod#>.64/29 Infrastructure: 172.16.<pod#>.60/30
Infrastructure: 172.16.<pod#>.64/29

podX-rtr1
Infrastructure
SW1 Rtr1
Potential Exfiltration traffic by
compromised infrastructure
router/s in 172.16.<pod#>.60/30 podX-rtr2
or 172.16.<pod#>.64/29
205
Task 9, 10, 11 – Network Forensics
• Task 9: Data collection
• Confirm that podX-rtr1 (ingress on port 0/0) and podX-rtr2 (ingree on port 0/1) are
exporting NetFlow v5 data to the NetFlow collector at 10.0.1.33 on UDP port 9995
• Confirm NetFlow data is being collected on podX-rtr1 and podX-rtr2 through the use of
show ip cache flow, show ip flow export commands.
• HTTP into the Kibana http://10.0.1.33/#/dashboard/file/podXX.json The dashboard
should be pre-populated with panels.

207
Task 9, 10, 11 – Network Forensics
• Task 10: In Kibana, use Filters and Queries to view
• DNS traffic and observe its characteristics.
• TCP traffic and observe its destinations
• ICMP traffic and observe suspicious packet count discrepancies and ICMP types/codes
• Suspicious UDP packets destined to 172.16.X.60-172.16.X.68.
• Suspicious inbound unidirectional traffic. Could it be C&C?
• Task 11: In Kibana, use Filters and Queries to view
• HTTP traffic
• Suspicious GRE packets
• Suspicious ISAKMP and Teredo traffic
• Suspicious outbound flows to 192.168.200.0/24? Could it be exfiltration?

208
Call to Action
• Visit the World of Solutions for
• Cisco Campus
• Walk in Labs
• Technical Solution Clinics

• Meet the Engineer

• Lunch and Learn Topics
• DevNet zone related sessions

222
Complete Your Online Session Evaluation
• Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.

• All surveys can be completed via

the Cisco Live Mobile App or the
Communication Stations

223
Thank you

224