Вы находитесь на странице: 1из 119

Troubleshooting GETVPN Deployments

BRKSEC-3051

Wen Zhang - Technical Leader, Services

2
Agenda

GETVPN Solution Overview


What Is GETVPN and Where Does It Fit?
Introduction to GETVPN
Technology Overview
GETVPN Deployment
Configuration and Deployment Considerations
Troubleshooting
Troubleshooting Tools and Techniques
Common Troubleshooting Scenarios

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Other Related Sessions

CiscoLive 2013

BRKSEC-2054 Deploying GET to Secure VPNs


BRKSEC-3013 Advanced IPSec with FlexVPN
BRKSEC-3052 Troubleshooting DMVPNs
BRKSEC-4054 Advanced Concepts of DMVPN

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
GETVPN Solution Overview
Cisco Group Encrypted Transport - GETVPN
What Is GETVPN?
Cisco GETVPN delivers a revolutionary solution for tunnel-less, any-
to-any and confidential branch communication
Large-scale any-to-any encrypted
communication
Any
Any --to
to --Any
Any Native routing without
Connectivity
Connectivity tunnel overlay
Optimal for QoS and Multicast
support - improves application
Cisco GET performance
VPN Transport agnostic - private
Scalable Real Time LAN/WAN, FR/ATM, IP, MPLS

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Tunnel-Less VPN - A New Security Model
Before: IPSec P2P Tunnels After: Tunnel-Less VPN

WAN

Multicast
Scalabilityan issue (N^2 problem) Scalable architecture for any-to-any
Overlay routing connectivity and encryption
Any-to-any instant connectivity cant No overlaysnative routing
be done to scale Any-to-any instant connectivity
Limited QoS Enhanced QoS
Inefficient Multicast replication Efficient Multicast replication
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
VPN Technology Positioning
Data Center Core

IPSec Agg.
GM
GM

Internet
Edge KS KS

Remot Access SW WAN Edge


Clients
Internet/Shared GET
Network MPLS/Private
Encrypted
Network

EzVPN/FlexVP GM GM GM
N Spoke
DMVPN/FLexVPN DMVPN/FlexVP
N Spoke GETVPN GM GETVPN GM GETVPN GM
Spoke
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
VPN Technology Positioning (Cont.)

FlexVPN DMVPN GETVPN

Public Internet Public Internet


Infrastructure Network Private IP Transport
Transport Transport
Hub-Spoke and
Converged Site to Site Any-to-Any;
Network Style Spoke-to-Spoke; (Site-
and Remote Access (Site-to-Site)
to-Site)
Dynamic Routing or
Dynamic routing on Dynamic routing on IP
Routing IKEv2 Route
tunnels WAN
Distribution
Route Distribution Route Distribution Route Distribution
Failover Redundancy
Server Clustering Model Model + Stateful

Peer-to-Peer Peer-to-Peer
Encryption Style Group Protection
Protection Protection

Multicast replication at Multicast replication at Multicast replication in


IP Multicast
hub hub IP WAN network

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Introduction to GETVPN
Group Encrypted Transport (GETVPN)

Uses three main components


Secure Group Keys
Header Preservation
Key Service
Is based on open standards with patented Cisco technology
Leverages existing IKE, IPSec, and multicast technologies
Takes advantage of the existing routing infrastructure

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
Group Security Functions
Routing Member
Key Server Key Server
Forwarding
Validate Group Members Replication
Manage Security Policy Routing
Create Group Keys
Distribute Policy/Keys

GM
Routing
Members
GM

GM
Group Member
Encryption Devices GM
Route Between Secure/ Unsecure Regions
Multicast Participation
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Group Security Elements
Group Policy KS Cooperative
Key Servers Protocol
Key Encryption Key (KEK)

Traffic Encryption Key (TEK)


GM
Routing
Members
GM

GM
RFC3547:
Group Domain of GM
Interpretation (GDOI)
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Basic GETVPN Architecture

Step 1: Group Members (GM) register via GDOI with the Key
Server (KS)

KS authenticates and authorizes the GM


KS pushes a set of IPSec SAs GM3 GM4
for the GM to use GM2

GM5

GM1

GM6

GM9
Key Server
GM8 GM7

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Basic GETVPN Architecture

Step 2: Data Plane Encryption


GM exchange encrypted traffic using the group keys
The traffic uses IPSec Tunnel Mode with Header Preservation
GM3 GM4

GM2
GM5

GM1
GM6

GM9

Key Server
GM8 GM7

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Basic GETVPN Architecture

Step 3: Periodic Rekey of Keys

KS pushes out replacement IPSec keys before current IPSec keys expire; this
is called a Rekey
GM3
GM4

GM2
GM5
GM1
GM6

GM9
Key Server
GM8
GM7

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Header Preservation
IPSec Tunnel Mode vs. GETVPN
IP Packet IP Header IP Payload

IPSec New IP Header ESP IP Header IP Payload


Tunnel Mode
IPSec header inserted by VPN Gateway
New IP Address requires overlay routing

IP Header IP Payload
IP Packet

GETVPN Preserved Header ESP IP Header IP Payload

IP header preserved by VPN Gateway


Preserved IP Address uses original routing plane

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
GETVPN Data Path
Host1 GM 1 GM2 Host2

Encrypted/Authenticated Using Group SA

Original Src and Original IP


Dst Addresses ESP Data
Header

Encrypted

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Rekey Methodology: Multicast Rekey

Rekey Message sent from key server to all group members


IP multicast message provides very efficient distribution
Rekeys resulting from configured KEK and TEK intervals Single rekey
packet sent to
or KS policy change multicast
enabled core
Key Server
GM2
GM1

Core replicates
the packets to
all GMs

GM3 GM4

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Rekey Methodology: Unicast Rekey

Key Server maintains state of active group members


Group Member sends ACK in response to the rekey messages
Remove Group Member if the GM does not acknowledge
three rekeys
Key Server
GM2
GM1

GM3 GM4

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Requirement for Time-Based Anti-Replay

Sequence number based anti-replay only works with


single sender
Need method to work for all senders using same IPSec SA
Key Server downloads relative pseudotime and window size to
all the GMs
GMs calculate pseudo-timestamp based on downloaded pseudotime and
sends out packet
Receiving GM verifies packet within window size
KS periodically refreshes GMs with pseudotime/window size - this means
clock does not need to be synchronized between GMs

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Time-Based Anti-Replay

If Senders pseudotime falls in the below Receiver window,


packet accepted

Reject Accept Reject

Initial PTr - W PTr PTr + W


pseudotime
Anti-replay window

Packet 1 and Packet 2 have pseudotimeT0, providing loose anti-replay


protection (unlike counter-based)

T0 T10 T20
Packet1
Packet2

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Cooperative Key Servers - HA
Single KS is a single point of failure
Two or more KSs known as COOP KSs manage a common set of keys and
security policies for GETVPN group members
Group members can register to any one of the available KSs

Cooperative KS1 Cooperative KS2

Subnet 1
Subnet 2

GM 1
GM 2
IP Network
Subnet 4 Subnet 3

GM 4 GM 3
GDOI Registration
Cooperative KS3
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Cooperative Key Servers (Cont.)
One KS is elected as the Primary KS
Cooperative KSs periodically exchange and synchronize groups database,
policy and keys
Primary KS is responsible to generate and distribute group keys

Cooperative KS1 Cooperative KS2 (Secondary)


(Primary)
Subnet 1
Subnet 2
GM 1
GM 2

IP Network
Subnet 4 Subnet 3
GM 4
GM 3
Announcement Messages
Rekey Messages Cooperative KS3 (Secondary)
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
GETVPN Deployment Configuration
COOP Server Exportable RSA Keys

RSA Keys (generated only on KSs) are required for rekey


authentication

RSA public key distribution from Key Server to Group Member:


Public key generated in the RSA key pair, is sent to the GM at the
registration
The rekeys are signed by the private key of the KS and GM verifies the
signature in the re-key with the public key of the KS
Exporting RSA Key between Key Servers:
One of the key server in the redundancy group should generate the
exportable RSA keys and copy those keys to other
key servers

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
KS Configuration
Pre-shared Key crypto keyring gdoi1
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
!
ISAKMP Policy crypto isakmp policy 10
encr 3des
IPSec Transform authentication pre-share
!
IPSec Profile crypto ipsec transform-set 3DES-SHA esp-3des esp-sha-hmac
!
crypto ipsec profile gdoi1
Access-List used for set security-association lifetime seconds 7200
defining set transform-set 3DES-SHA
rekey (useful in !
multicast rekeys only) access-list 150 permit ip any host 225.1.1.1
!
access-list 160 deny eigrp any any
Access-list defining the access-list 160 deny pim any any
encryption policy access-list 160 deny udp any any eq isakmp
access-list 160 deny udp any any eq 848
access-list 160 permit ip any any

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
KS Configuration (Cont.)
GDOI Group ID crypto gdoi group getvpn1
identity number 101
Rekey Address mapping server local
(only for multicast rekeys) !rekey address ipv4 150 !
rekey lifetime seconds 14400
rekey retransmit 10 number 2
Rekey Properties rekey authentication mypubkey rsa getvpn1
rekey transport unicast

sa ipsec 1
Encryption ACL
profile gdoi1
match address ipv4 160
Source address for rekeys address ipv4 130.23.1.1
redundancy
local priority 10
COOP KS Config
peer address ipv4 130.1.2.1
!

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
GM Configuration
Pre-shared Key crypto keyring gdoi
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123
!
ISAKMP Policy
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
GDOI Group !
crypto gdoi group getvpn1
KS Address identity number 101
server address ipv4 130.23.1.1
GDOI crypto map !
crypto map getvpn10 gdoi
set group getvpn1
Crypto map on the !
interface interface FastEthernet0/0
crypto map getvpn

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
GETVPN Platform Support

Platform Group Member Key Server


Software Yes Not supported
870 Yes Not supported
1821 Yes Not supported
1841/1900 Yes Yes
2800 (AIM/SSL)/2900 Yes Yes
3800 (AIM-II/AIM-III)/3900 Yes Yes
7200 NPEG1, VAM2+ Yes Yes
7301 NPEG1, VAM2+ Yes Yes
7200 NPEG2, VAM2+ Yes Yes
7200 NPEG2, VSA Yes Yes
Cisco ASR 1000 Yes Yes (since XE3.6)

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Scalability and Performance

GETVPN Provides complete segregation of control and data plane


Key Server is responsible to maintain the control plane (key
management) and GM is responsible to handle the data plane (actual
user traffic)
KS and GM can NOT be configured on same IOS device
KS should be properly sized for number of branches (scale) in the
network
GM should be properly sized for traffic throughput at each branch

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Deployment Best Practices
IKE/IPSec
Use specific pre-shared keys for all the GMs and KSs instead of using default key
KS
Always use COOP KSs
Set the huge buffer to 65535 and add 10 buffers to permanent buffer list
Configure periodic DPDs between the COOP KSs
Enable GM authorization
Policy
Aggregate the permit access-list entries to reduce the entries
Enable Time-Based Anti-Replay
Avoid re-encrypting traffic which is already encrypted (SSH, HTTPS)
Registration
Distribute GM registration to multiple KSs by arranging the KS order in configuration
Rekey Timers
Set TEK lifetime to 7200 Seconds
Set KEK lifetime to 86400 Seconds

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
GETVPN Troubleshooting
A problem well stated is a problem half solved

Charles F. Kettering
Troubleshooting GETVPN

Ultimately all problems manifest at the data plane -my user


application is not working over GETVPN!
But where really is the problem?
Control Plane
Events that lead up to SAs getting installed on the GMs
Data plane
Policy downloaded with SAs installed but traffic is not flowing

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Troubleshooting GETVPN High Level Flow

Time Based
COOP Anti-Replay

IKE Fragmentation
MTU Issues
Registration
Transport Issues
Policy Download
Crypto
Rekey
policy/engine

Control Plane Data Plane

Troubleshooting Flow
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
GETVPN Control Plane

Common Control Plane Issues


GM registration issues
Policy download issues
COOP issues
Rekey failures

Understand the expected protocol flow and know how to check for them

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Control Plane Troubleshooting Tools
GETVPN provides enhanced set of show commands for
functionality verification
IOS also provided wide variety of syslog messages to verify
proper GETVPN operations, and early insight into potential
problems
IPSec and GDOI related debugs can then be enabled for further
troubleshooting
GDOI conditional debugs 15.1(3)T
GDOI event trace 15.1(3)T

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Show crypto gdoi (on KS)
Group Name : GET
Group Identity : 101
Registered GMs Group Members : 3
IPSec SA Direction : Both
Active Group Server : Local
COOP configuration Redundancy : Configured
Local Address : 130.23.1.1
Local Priority : 10
Local KS Status : Alive
Key Server Role Local KS Role : Primary
Group Rekey Lifetime : 1800 secs
Group Rekey
KEK lifetime remaining Remaining Lifetime : 88 secs
Rekey Retransmit Period : 10 secs
Rekey Retransmit Attempts: 3
Group Retransmit
Remaining Lifetime : 0 secs

IPSec SA Number : 1
IPSec SA Rekey Lifetime: 900 secs
Profile Name : gdoi1
Replay method : Count Based
Replay Window Size : 64
SA Rekey
TEK lifetime remaining Remaining Lifetime : 446 secs
ACL Configured : access-list 160

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Show crypto gdoi ks member (on KS)
KS#show crypto gdoi ks members

Group Member Information :

Number of rekeys sent for group GET: 4

GMs IP address Group Member ID : 131.1.1.1


Group ID : 101
Group Name : getvpn1
KS GM is registered with Key Server ID : 130.2.1.1
Rekeys sent : 4
Rekey Acks Rcvd : 4

GM rekey history Sent seq num : 1 2 3 4


Rcvd seq num : 1 2 3 4

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
Show crypto gdoi (on GM)
GROUP INFORMATION KEKPOLICY:
Rekey Transport Type : Unicast
Group Name : GET Lifetime (secs) : 12295
Active KS
Group Identity : 101 Encrypt Algorithm : 3DES
Rekeys received : 270 Key Size : 192
IPSec SA Direction : Both Sig Hash Algorithm : HMAC_AUTH_SHA
Active Group Server : 134.50.0.1 Sig Key Length (bits) : 1024
Group Server list : 134.50.0.1
TEK POLICY:
FastEthernet0/0:
GM Reregisters in : 5187 secs
Rekey Received(hh:mm:ss) : 00:02:30 IPSec SA:
sadirection:outbound
Rekeys received spi: 0x7C45C74A(2084947786)
Cumulative : 270 transform: esp-aes esp-sha-hmac
After registration : 270 When was sa timing: remaining key lifetime
Rekey Acks sent : 270 last rekey (sec): (5246)
received Anti-Replay(Time Based) : 2 sec interval
ACL Downloaded From KS 134.50.0.1:
access-list deny eigrp any any Remaining
access-list deny tcp any any port = 179 IPSec SA
access-list deny udp any port = 848 Lifetime
any port = 848
access-list permit ip any any

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
GETVPN Control Plane Verification
Syslog Messages - KS

Rekey:
GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1
from address 101.1.1.1 with seq # 1

COOP:
GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.0.9.1 Unreachable in group
G1
GDOI-5-COOP_KS_ELECTION: KS entering election mode in group G1
(Previous Primary = NONE)
GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.0.8.1 in group G1 transitioned to
Primary (Previous Primary = NONE)

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
GETVPN Control Plane Verification
Syslog Messages - GM

Registration:
CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1
using address 10.1.13.2
GDOI-5-GM_REKEY_TRANS_2_UNI: Group G1 transitioned to Unicast Rekey
GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for group
G1 using address 10.1.13.2

Rekey:
GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to
10.1.13.2 with seq # 3

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
Control Plane Debugging Challenges
Challenge
Networks are getting bigger and faster, traditional debugs may not scale
Solution
Use IPSec and GDOI conditional debugs to minimize the debugging
impact
Use the minimal level of debugs required
Challenge
Problems can be unpredictable with no identifiable trigger
Solution
Syslogs
GDOI Event Trace
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
GDOI Debug Level Granularity

All feature components can be debugged at 5 levels


Start with the highest level, enable additional levels as needed
GM1#debug crypto gdoi gm rekey ?
all-levels All levels
detail Detail level
error Error level
event Event level
packet Packet level
terse Terse level

Debug Level What you will get


Error Error Conditions
Terse Important messages to the user and protocol issues
Event State transitions and events such as send/receive rekeys
Detail Most detailed debug message information
Packet Dump of detailed packet information
All All of the above
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
GDOI Conditional Debugs
All IPSec and GDOI debugs can now be triggered with KS1 KS2

a conditional filter based on group or peer address


Use the unmatched flag to catch debugs with no MPLS/Private IP
context information GM1 GM500

To enable conditional debugs


1) Set the conditional filter
?
GM145

2) Enable relevant debugs of interest as usual


KS1# debug crypto gdoi condition peer add ipv4 10.1.20.2
% GDOI Debug Condition added.

KS1#
KS1# show crypto gdoi debug-condition
GDOI Conditional Filters:
Peer Address 10.1.20.2
Unmatched NOT set

KS1#debug crypto gdoi ks registration all-levels


GDOI Key Server Registration Debug level: (Packet, Detail, Event, Terse, Error)

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Best practices when using the debug commands

Turn off console logging


Use NTP to sync up times on all devices
Enable msectimestamping debug and log messages
service timestamps debug datetime msec
service timestamps log datetime msec
Send the debugs to a syslog server
If no syslog server is available, use the logging buffer with an increased
buffer size
logging buffered 1000000 debugging
terminal exec prompt timestamp when using the show commands to
correlate show commands with the debug output
reload in x to prepare for the worst

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
GDOI Event Trace

Light weight event buffer to supplement syslogs


Always-on
Flexible output and display options
Event buffer
Continuous real time output
Output to file
Merged output from different feature components
Circular or one-shot buffer
Extensive exit path/error tracing capability

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
GDOI Event Trace - Example
GM1#show monitor event-trace gdoi?
all Show all the traces in current buffer
back Show trace from this far back in the past
clock Show trace from a specific clock time/date
coop GDOI COOP Event Traces
from-boot Show trace from this many seconds after booting
infra GDOI INFRA Event Traces
latest Show latest trace events since last display
merged Show entries in all event traces sorted by time
registration GDOI Registration event Traces
rekey GDOI Rekey event Traces

GM1#show monitor event-trace gdoi merged all


*May 25 20:20:57.706: Registration_events: GDOI_REG_EVENT: REGISTRATION_STARTED:
GM 10.1.20.2 to KS 10.1.11.2 for group G1
*May 25 20:21:08.970: Registration_events: GDOI_REG_EVENT: REGISTRATION_DONE: GM
10.1.13.2 to KS 10.1.11.2 for group G1
*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: REKEY_RCVD: From 10.1.11.2
to 10.1.13.2 with seq no 131 for the group G1
*May 26 00:45:52.878: Rekey_events: GDOI_REKEY_EVENT: ACK_SENT: From 10.1.11.2
to 10.1.13.2 with seq no 131 for the group G1

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Troubleshooting Methodology
crypto gdoi group G1 crypto gdoi group G1
identity number 3333 identity number 3333
server local server local
rekey lifetime seconds 86400 rekey lifetime seconds 86400
rekey authenmypubkeyrsa get rekey authenmypubkeyrsa get
rekey transport unicast rekey transport unicast
sa ipsec 1 KS2 saipsec 1
KS1
profile gdoi-p profile gdoi-p
match address ipv4ENCPOL match address ipv4ENCPOL
replay counter window-size 64 replay time window-size 5
address ipv4 10.1.11.2 address ipv4 10.1.12.2
Ser 1/0: 10.1.11.2 Ser 1/0: 10.1.12.2
redundancy redundancy
local priority 10 local priority 2
peer address ipv4 10.1.12.2 MPLS/Private IP peer address ipv4 10.1.11.2

Ser 1/0: 10.1.20.2 Ser 1/0: 10.1.21.2


crypto gdoi group G1 crypto gdoi group G1
identity number 3333 identity number 3333
server address ipv4 10.1.11.2 server address ipv4 10.1.12.2
server address ipv4 10.1.12.2 GM1 server address ipv4 10.1.11.2
GM2
! !
crypto map gm_map 10 gdoi crypto map gm_map 10 gdoi
set group G1 Eth 0/0: 192.168.20.1/24 set group G1
Eth 0/0: 192.168.21.1/24
! !
interface Serial1/0 interface Serial1/0
crypto map gm_map crypto map gm_map

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
GETVPN Control Plane Setup Steps

COOP KS IKE Setup


COOP Election and Policy Creation

GM-KS IKE Setup


GM Authorization and Registration
GM Encryption Keys and Policy download
GM Data Encryption and Decryption

Periodic Key Renewal and Distribution (Rekeys)

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
GETVPN Common Issues Control Plane

IKE Setup

Encryption Policy

Key RenewalRekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
IKE Setup Between KS and GM
First step in GM registration is IKE setup
On successful negotiation of the IKE process, GM proceeds with the
GDOI group registration
IKE SA is established at the time of registration eventually times out
as its no longer needed after registration
KS1# show crypto isakmp sa
IPv4 Crypto ISAKMP SA
Dst src state conn-id slot status
10.1.11.2 10.1.20.2 GDOI_IDLE 1013 0 ACTIVE
10.1.12.2 10.1.11.2 GDOI_IDLE 1004 0 ACTIVE
10.1.21.2 10.1.11.2 GDOI_REKEY 0 0 ACTIVE
Expires
after IKE
lifetime
GM1# show crypto isakmp sa
IPv4 Crypto ISAKMP SA
Dst src state conn-id slot status
10.1.11.2 10.1.20.2 GDOI_IDLE 1073 0 ACTIVE
10.1.20.2 10.1.11.2 GDOI_REKEY 1074 0 ACTIVE

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
IKE Setup IKE Failure
Symptoms

If a GM fails to register with the KS, it will continue to attempt


to register with the KS
*May 24 06:40:15.581: %CRYPTO-5-GM_REGSTER: Start registration to KS
10.1.11.2 for group G1 using address 10.1.20.2
GM1#
*May 24 06:41:25.581: %CRYPTO-5-GM_REGSTER: Start registration to KS
10.1.11.2 for group G1 using address 10.1.20.2

%GDOI-5-GM_REGS_COMPL: Registration to KS 10.1.11.2 complete for


group G1 using address 10.1.20.2

KS1 KS2
Possible causes:
Network issues between the GM and KS MPLS/Private IP

IKE negotiation failure


KS policy issues GM1 GM2

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Pre-Shared Key Mismatch
Troubleshooting

Verify routing information on KS and GM and try ping KS from the GM


After ruling out the connectivity issues, check the IKE SA on the GM

GM1#show crypto isakmp sa


IPv4 Crypto ISAKMP SA
Dst src state conn-id status
10.1.11.2 10.1.20.2 MM_KEY_EXCH 1038 ACTIVE

IPv6 Crypto ISAKMP SA

IKE SA not getting established; cant


get to GDOI_IDLE state

Verify the logs on the Key Server


KS1#
%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 10.1.20.2 failed its
sanity check or is malformed

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Pre-Shared Key Mismatch
Solution

Syslog pointing to a mismatched pre-shared key configuration


Can be verified using debug crypto isakmp

KS Config: crypto isakmp key cicso address 10.1.20.2

GM Config: crypto isakmp key cisco address 10.1.11.2

Correct the pre-shared key configuration

KS1(config)#no crypto isakmp key cicso address 10.1.20.2


KS1(config)#crypto isakmp key cisco add 10.1.20.2
KS1(config)#^Z

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
GETVPN Common Issues Control Plane

IKE Setup

Encryption Policy

Key RenewalRekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
GM Policy Download

As part of the registration process, KS pushes down the


encryption policies and keying material to the GM:
GM1#show crypto gdoi
<snip>
ACL Downloaded From KS 10.1.11.2:
access-list deny eigrp any any
access-list deny ip 224.0.0.0 0.0.0.255 any
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
access-list permit ip any any

KEK POLICY:
Rekey Transport Type : Unicast
Lifetime (secs) : 2954
<snip>
TEK POLICY:
Serial1/0:
IPSec SA:
sa direction:inbound
spi: 0x2113F73B(554956603)
transform: esp-3desesp-sha-hmac
sa timing:remaining key lifetime (sec): (99)
Anti-Replay(Time Based) : 5 sec interval
<snip>

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
KS Policy Issues
Routing Control Plane Traffic Failure

In most environments, GETVPN runs on the CE devices and KS1 KS2

PE devices do not participate in GETVPN


MPLS/Private IP

Failure to deny control plane traffic (such as routing protocol) BGP

on the PE-CE link will cause routing protocol to go down as GM1


GM2

soon as GM successfully registers


To identify, look at the ACL downloaded at GM:
GM1#show crypto gdoi gm acl
Group Name: G1
ACL Downloaded From KS 10.1.11.2:
access-list deny eigrp any any BGP is not denied in the ACL
access-list deny ip 224.0.0.0 0.0.0.255 any downloaded from the KS
access-list deny ip any 224.0.0.0 0.0.0.255
access-list deny udp any port = 848 any port = 848
access-list permit ip any any
ACL Configured Locally:

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
KS Policy Issues
Control Plane Traffic - Solution

If most of the CEs are running BGP with the PE routers, KS1 KS2

configure a global KS policy to deny BGP


MPLS/Private IP
KS1&2(config)# ip access-list extended ENCPOL BGP
KS1&2(config-ext-nacl)#1 deny tcp any any eq bgp
GM2
KS1&2(config-ext-nacl)#2 deny tcp any eq bgp any GM1

If only a handful of CEs are running BGP with the PE routers,


configure a local GM policy to deny BGP
GM1# GM1#show crypto gdoi gm acl
! Group Name: G1
access-list 150 deny tcp any any eq bgp ACL Downloaded From KS 10.1.11.2:
access-list 150 deny tcp any eq bgp any <snip>
! access-list permit ip any any
crypto map gm_map 10 gdoi ACL Configured Locally:
set group G1 Map Name: gm_map
match address 150 access-list 150 deny tcp any any port = 179
! access-list 150 deny tcp any port = 179 any

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
GETVPN Common Issues Control Plane

IKE Setup

Encryption Policy

Key Renewal - Rekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
GETVPN Rekeys

Once the GETVPN network is properly setup and is working, KS is


responsible for sending out rekey messages to all the GMs
KS can use unicast or multicast rekeys
Following syslog messages will appear in the log:

PRIMARY KS:

%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from


address 10.1.11.2 with seq # 11

All the GMs:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to


10.1.20.2 with seq # 11

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Following the Rekey Flow

Rekey received by
IP?

Rekey Rekey Rekey verified by


Rekey sent?
delivered? received? IKE?

KS Network GM
Transport Rekey Processed
by GDOI?

Rekey
Acknowledged?

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Missing RSA Key
Symptoms

When GM registers to the KS, the following KS1 KS2

message shows up in the syslog:


MPLS/Private IP

%GDOI-1-KS_NO_RSA_KEYS: RSA Key - get : Not found, Required for group G1 GM1 GM2

As a result KS will not send rekey messages, and GM


will re-register when the keys expire

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have


expired/been cleared, or didn't go through. Re-register to KS.

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Missing RSA Key on the KS
Troubleshooting Steps

Check whether KS is sending out the rekeys or not:

KS1#show crypto gdoi ks rekey


Group G1 (Multicast)
Number of Rekeys sent : 0
Number of Rekeys retransmitted : 0
KEK rekey lifetime (sec) : 86400
Retransmit period : 10 No rekeys sent
Number of retransmissions : 2
IPSec SA 1 lifetime (sec) : 3600
Remaining lifetime (sec) : 166
Number of registrations after rekey : 22

KS needs RSA keys to sign the rekey messages; check logs for
clues and/or verify the RSA keys

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Missing RSA Key on the KS
Troubleshooting Steps (Cont.)

Verify RSA key configuration on the KS:


KS1#show running | section gdoi group
crypto gdoi group G1
identity number 3333
server local
rekey address ipv4 102
rekey lifetime seconds 86400
rekey authentication mypubkey rsa get
sa ipsec 1
profile gdoi-p
match address ipv4ENCPOL Labeled RSA key not present
no replay
address ipv4 10.1.11.2

Verify the RSA key pair name on the router:


KS1#show crypto key mypubkey rsa | include name
Key name: key1
Key name: key1.server

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Missing RSA Key on the KS
Solution

Generate the required RSA key pair


KS1(config)#crypto key generate rsa label get exportable modulus 1024
The name for the keys will be: getvpn-rsa-key

% The key modulus size is 1024 bits


% Generating 1024 bit RSA keys, keys will be exportable...[OK]

Verify rekey messages are now being sent on the KS


%GDOI-5-KS_SEND_UNICAST_REKEY: Sending Unicast Rekey for group G1 from
address 10.1.11.2 with seq # 1

KS1#show crypto gdoi ks rekey


Rekeys are now sent
Group G1 (Unicast)
Number of Rekeys sent : 1
<SNIP>

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Multicast Rekey Issues
Multicast Rekeys Failing - Symptom

GM is not getting the multicast rekey messages and therefore


continues to re-register with the KS
Rekey starts to work when switched from multicast rekey to
unicast rekey
Possible Causes
Packet delivery issue within the multicast routing infrastructure
End-to-end multicast routing enabled?
mVPN service provided by the MPLS core provider?

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Multicast Rekey Failing
Troubleshooting

Check KS to verify multicast rekey messages are being sent KS1 KS2

Multicast
Network
%GDOI-5-KS_SEND_MCAST_REKEY: Sending Multicast Rekey for group G1
10.1.20.2 10.1.21.2
from address 10.1.11.2 to 226.1.1.1 with seq # 6
GM1 GM2

Make sure ICMP is excluded from the KS encryption


policy and is used as a tool to test multicast

KS1#ping 226.1.1.1

Type escape sequence to abort.


Sending 1, 100-byte ICMP Echos to 226.1.1.1, timeout is 2 seconds:

Reply to request 0 from 10.1.21.2, 44 ms


No response from
GM1 (10.1.20.2)

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Multicast Rekey Failing
Troubleshooting

Check the multicast forwarding path

WAN#show ip mroute 226.1.1.1


<snip>
(10.1.11.2, 226.1.1.1), 00:13:18/00:02:56, flags: T
Incoming interface: Serial0/0, RPFnbr 0.0.0.0
Outgoing interface list:
Serial3/0, Forward/Sparse-Dense, 00:13:18/00:00:00
Verify the OIL

Check the PIM neighbor


WAN#sh ip pim neighbor
PIM Neighbor Table
Neighbor Interface Uptime/Expires Ver DR
Address Prio/Mode
10.1.11.2 Serial0/0 01:03:54/00:01:16 v2 1 / S
10.1.21.2 Serial3/0 01:13:06/00:01:26 v2 1 / S

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
Multicast Rekey Failing
Solution

Enable PIM on the WAN router towards the GM

WAN(config)#int s2/0
WAN(config-if)#ip pim sparse-dense-mode
WAN(config-if)#end

%PIM-5-NBRCHG: neighbor 10.1.20.2 UP on interface


Serial2/0 (vrf default)

Check multicast routing path again


Re-test with multicast ping
Verify GM now receives the multicast rekey messages

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Unicast Rekey Failing
Transient Network Issues

Due to transient changes in the network, unicast rekey packets might not
make it to the GM(s)
If the GMs does not receive the rekey, it will have to re-register

Symptoms:

Missing Following syslog on GM:

%GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from 10.1.11.2 to 10.1.21.2


with seq # 3

GM shows re-registration syslog:

%GDOI-4-GM_RE_REGISTER: The IPSec SA created for group G1 may have expired/been


cleared, or didn't go through. Re-register to KS.
%CRYPTO-5-GM_REGSTER: Start registration to KS 10.1.11.2 for group G1 using
address 10.1.20.2

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Unicast Rekey Failing
Troubleshooting and Solution

Verify whether the rekeys are not being sent, not being received
or not being processed
KS: GM:
show crypto gdoi ks members show crypto gdoi gm rekey

Group Member Information : Group G1 (Unicast)


Number of rekeys sent for group G1 : 380 Number of Rekeys received (cumulative) : 0
Number of Rekeys received after registration : 0
Group Member ID : 10.1.20.2 Number of Rekey Acks sent : 0
Group ID : 3333 Rekey (KEK) SA information :
Group Name : G1 dstsrcconn-id my-cookie his-cookie
Key Server ID : 10.1.11.2 New : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61
Rekeys sent : 1 Current : 10.1.20.2 10.1.11.2 1098 44F7FC328302AC61
Rekeys retries : 0 Previous: --- --- --- --- ---
Rekey Acks Rcvd : 0
Rekey Acks missed : 0

Always configure retransmissions to overcome transient issues


Unicast rekey dropped
rekey retransmit 30 number 3
Make sure UDP port 848 is not blocked in the data path
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Rekey Fails Signature Validation

Primary KS fails, GM receives rekey from secondary KS, but


receives error:
*Apr 27 18:18:19.511: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode
failed with peer at 10.1.12.2

Syslog is not conclusive, lets see what we can get with some debugs
Signature validation failed!
GM1# debug crypto isakmp
Crypto ISAKMP debugging is on
GM1#
GM1# debug crypto gdoi
GDOI Generic Debug level: (Error, Terse)
*Apr 27 18:18:19.251: ISAKMP (0:1014): received packet from 10.1.12.2 dport 848
sport 848 Global (R) GDOI_REKEY
*Apr 27 18:18:19.251: GDOI:INFRA:(G1:0:1014:HW:0):Received Rekey Message!
*Apr 27 18:18:19.259: GDOI:INFRA:(G1:0:1014:HW:0):Signature Invalid! status = 13
*Apr 27 18:18:19.259: %CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed
with peer at 10.1.12.2
*Apr 27 18:18:19.259: ISAKMP: Receive GDOI rekey: Processing Failed. IKMP error = 6

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Rekey Fails Signature Validation
Solution

Problem:
Secondary KS has its own RSA key pair instead of the
exported key pair from the primary KS1 KS2

To verify, compare the RSA key pairs


MPLS/Private IP

KS#show crypto key mypubkey rsa


GM1 GM2

Solution:
Generate exportable RSA key pair on the primary KS

KS1(config)#crypto key generate rsa modulus 1024 exportable label key1

Export RSA key pair to all secondary KSs


KS2(config)#crypto key import rsa key1 pem terminal <passphrase>

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
GETVPN Common Issues Control Plane

IKE Setup

Encryption Policy

Key RenewalRekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Control Plane Replay Check Detection

Control Plane messages can carry time sensitive information and therefore
require replay protection
Rekey messages from KS to GM
COOP Announcement messages between KSs
Sequence number check to protect against replayed messages
Pseudotime check to protect against delayed messages with TBAR
enabled
Control Plane Replay check added in IOS version 12.4(15)T10, 12.4(22)T3,
12.4(24)T2, 15.0(1)M, and later

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Control Plane Replay Check
Code interoperability issue

Problem: customer upgraded IOS on a GM to 15.0(1)M for a bug fix,


and started to experience KEK rekey failures
The following errors are observed in the syslog

%GDOI-3-GDOI_REKEY_SEQ_FAILURE: Failed to process rekey seq # 1 in seq payload


for group G1, last seq # 11
%GDOI-3-GDOI_REKEY_FAILURE: Processing of REKEY payloads failed on GM 10.1.13.2
in the group G1, with peer at 10.1.11.2
%CRYPTO-6-IKMP_MODE_FAILURE: Processing of GDOI mode failed with peer at
10.1.11.2

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Control Plane Replay Check
Code interoperability issue - solution

KS does not support control plane replay detection, and resets the rekey
sequence # for KEK rekey
GM interprets that as a replayed rekey message
Solution is to upgrade the KS to an IOS version that also supports the
control plane replay detection
New behavior KEK Rekey

*Apr 6 15:41:26.932: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from


10.1.11.2 to 10.1.13.2 with seq # 8
GM1#
*Apr 6 15:42:01.940: %GDOI-5-GM_RECV_REKEY: Received Rekey for group G1 from
10.1.11.2 to 10.1.13.2 with seq # 1

TEK Rekey with seq# reset

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Control Plane Replay Check IOS Upgrade procedure

Recommended IOS releases


IOS: 15.2(4)M3
IOS-XE: 15.1(3)S4

IOS upgrade procedure


Step 1. Upgrade a secondary KS first, wait until COOP KS election is
completed
Step 2. Repeat step 1 for all secondary KS
Step 3. Upgrade primary KS
Step 4. Upgrade Group Members

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
GETVPN Common Issues Control Plane

IKE Setup

Encryption Policy

Key RenewalRekey

Control Plane Replay Check

Control Plane Packet Fragmentation Issue

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Control Plane Fragmentation Issues
COOP Announcement Packets

In a large network (1500+ GMs), COOP update packet becomes larger


than the default maximum buffer size
Default huge buffer size is 18024 bytes
Syslog message appears on the KSs:

%SYS-2-GETBUF: Bad getbuffer, bytes= 18872 -Process= "Crypto IKMP", ipl= 0, pid= 183

Tune buffers to increase huge buffers and add buffers to permanent list:
buffers huge permanent 10
buffers huge size 65535

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Control Plane Fragmentation Issues (cont.)
COOP Announcement Packets

Large ANN messages are fragmented in transit between KSs


Can have up to 40+ IP fragments KS1 Frag
1
KS2

Frag2
One dropped fragment -> entire ANN dropped Frag3

How to identify? Frag4

%GDOI-3-COOP_KS_UNREACH: Cooperative KS 10.1.11.1 Unreachable in group G1.


%GDOI-5-COOP_KS_TRANS_TO_PRI: KS 10.1.12.2 in group G1 transitioned to FragN
Primary (Previous Primary = 10.1.11.2)

KS1#show ip traffic | section Frags


Frags: 10 reassembled, 3 timeouts, 0 couldn't reassemble
0 fragmented, 0 fragments, 0 couldn't fragment

Need to look at transit path features that may drop fragments,


Firewall, VFR, reassembly buffer size, etc.

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Troubleshooting GETVPN Data Plane

Ultimately all problems manifest at the data plane -my user


application is not working over GETVPN!
But where really is the problem?
Control Plane
Events that lead up to SAs getting installed on the GMs
Data plane
Policy downloaded with SAs installed but traffic is not flowing

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Generic IPSec Data Plane Troubleshooting

Need to have complete understanding of the forwarding path and how


to checkpoint it
Which device is the culprit, encrypting or decrypting router?
In which direction is the problem happening, ingress or egress?
Some syslogs may help reveal data plane drops
Data plane errors are typically rate limited
Common errors include replay, authentication failures
Heavily dependent upon show commands and counters to trace the
packet path
Sniffer capture of limited use due to encryption, however
ESP-NULL same crypto processing except packets not encrypted
DSCP coloring of packets to uniquely identify a flow

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
GETVPN Data Plane

IPSec tunnel mode just like IPSec classic so most IPSec troubleshooting
techniques still apply, however
Symmetrical encryption policy requirement
Unique challenges with Header Preservation
PMTUD
Time Based Anti-Replay
Extra encapsulation overhead Fragmentation boundary condition calculation
Timer Based Anti-Replay failure

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
Data Plane Troubleshooting Tools

Interface counters
Encryption/decryption counters
Netflow
IP Accounting
ACL
DSCP packet coloring
Embedded Packet Capture (EPC)

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
IPSec Data Plane Packet Flow Checkpoints
3 4
Traffic Direction 1 6

GM1
Private WAN
GM2

Client Server
2 5
Encrypting GM Decrypting GM
1. Ingress LAN interface 4. Ingress WAN interface
Input ACL Input ACL
Ingress Netflow Ingress Netflow
Embedded Packet Capture
Embedded Packet Capture
2. Crypto engine Input IP precedence accounting
show crypto ipsec sa
5. Crypto engine
show crypto session detail show crypto ipsec sa
3. Egress WAN interface show crypto session detail
Egress Netflow 6. Egress WAN interface
Embedded Packet Capture Egress Netflow
Output IP precedence accounting
Embedded Packet Capture

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Importance of a Controlled Test

The case for ping x.x.x.x timeout 0


Separation from background traffic
Poor mans conditional filter
Packet coloring/marking
Tools to monitor based on DSCP/Precedence marking
ESP-NULL
IP characteristics for seemingly application issues
Ping works but TCP doesnt?
Why does IPSec care about TCP, or does it?

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Encrypting GM Data Plane Flow

Verify clear traffic being received with Ingress Netflow


interface Ethernet0/0
ip address 192.168.13.1 255.255.255.0
ip flow ingress
!
GM1# show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Et0/0 192.168.13.2 Se1/0 192.168.14.2 06 E443 0017 11

Verify encryption operation performed


TCP port 23 = telnet
Lack of per-flow granularity
GM1# show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 4, origin: crypto map
Inbound: #pkts dec'ed 162 drop 0 life (KB/Sec) 0/146
Outbound: #pkts enc'ed 170 drop 0 life (KB/Sec) 0/146

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Encrypting GM Data Plane Flow Cont.

Verify encrypted traffic existing GM with egress Netflow


interface Serial/0
ip address 10.1.13.2 255.255.255.252
ip flow egress
! Protocol 50 = ESP
GM1#show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Et0/0 192.168.13.2 Se1/0* 192.168.14.2 32 EE5B 2BEF 170

GM1#show crypto ipsec sa


interface: Serial1/0 Active IPSec SA SPI
<snip>
current outbound spi: 0xEE5B2BEF(3998952431)

If per L4 flow granularity is desired, can use inbound precedence


coloring and egress precedence accounting

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
Decrypting GM Data Plane Flow
Verify encrypted traffic arriving on GM with Netflow
Protocol 50 = ESP
GM2#show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Se1/0 192.168.13.2 Et0/0 192.168.14.2 32 EE5B 2BEF 170

Verify traffic decryption Inbound IPSec SA SPI


GM2#show crypto session detail
<snip>
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 10, origin: crypto map
Inbound: #pkts dec'ed 170 drop 0 life (KB/Sec) 0/150
Outbound: #pkts enc'ed 162 drop 0 life (KB/Sec) 0/150

Verify clear traffic forwarding post decryption


TCP port 23 = telnet
GM2#show ip cache flow
<snip>
SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts
Se1/0 192.168.13.2 Et0/0* 192.168.14.2 06 E6CC 0017 170

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
GETVPN Common Issues Data Plane

Asymmetrical Encryption Policy

Fragmentation/Path MTU

Other data plane issues common to IPSec

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
KS Policy Issues
Data Plane Traffic Failure

Encryption policies (what needs to be encrypted) are defined centrally at the KS


Symmetrical ACLs should be defined to either permit
or to deny traffic from getting encrypted
If the traffic is not being encrypted or being blocked, verify we have symmetrical ACL

MPLS/Private IP

GM1 GM2
Ethernet 0/0: Ethernet 0/0:
192.168.20.0/24 192.168.21.0/24

KS Access-list
ip access-list extended ENCPOL
permit ip 192.168.20.0 0.0.0.255 192.168.21.0 0.0.0.255
permit ip 192.168.21.0 0.0.0.255 192.168.20.0 0.0.0.255

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
GETVPN Common Issues Data Plane

Asymmetrical Encryption Policy

Fragmentation/Path MTU

Other data plane issues common to IPSec

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 98
Fragmentation Issues
PMTU Discovery

Large packets with the DF bit set may get black-holed in the GETVPN network

MTU 1500 MTU 1500

GM1
MTU 1000 GM2

1400B 1460B

ICMP 3/4

Server sends a large packet with the DF bit set in an attempt to


perform network PMTUD

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
PMTUD and GETVPN

Encrypting GM adds IPSec overhead and forwards it


Intermediate router drops the packet and sends back icmp3/4 to perform
PMTUD, two possibilities
This ICMP dropped by the encrypt GM because its not encrypted based on the
encryption policy
This ICMP gets forwarded to the end host but gets dropped due to
unauthenticated payload
Bottom line: PMTUD does not work with the current header preservation
implementation of GETVPN

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 100
PMTUD and GETVPN

Solution
Implement ip tcp adjust-mss to reduce the TCP packet segment size
Clear the DF bit in the encapsulating header
User Traffic
Encrypting GM

DF=0 DF=0 Data

interface Ethernet0/0 DF=1 Data


ip address 192.168.13.1 255.255.255.0
ip policy route-map clear-df-bit
!
route-map clear-df-bit permit 10
match ip address 111
set ip df 0
!
access-list 111 permit tcp any any

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 101
GETVPN Common Issues Data Plane

Asymmetrical Encryption Policy

Fragmentation/Path MTU

Other Data Plane Issues Common to IPSec

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 102
IPSec drop due to packet corruption

IPSec integrity check makes IPSec packets a lot more sensitive to packet
corruption in the network
Packet corruption symptoms
%CRYPTO-4-RECVD_PKT_MAC_ERR: decrypt: mac verify failed for connection id=695
local=192.168.14.2 remote=192.168.13.2 spi=7C4E759F seqno=00000001

How to prove packets are corrupted in the network?


Enable EPC to capture packets into a circular buffer on both GMs
Use EEM (Embedded Event Manager) to
Synchronize and stop the capture on both routers when the RECVD_PKT_MAC_ERR message is
logged
Notify the network operator by email

Retrieve both captures to examine for packet corruption


BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 103
GETVPN Troubleshooting Summary

Have a clear and concise problem description


Try to break the problem down to either control or data plane
Understand the expected protocol flow on the control plane and how
to check for them
Understand where/how to checkpoint the data plane
Syslog and event trace your friend
There is always TAC!

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 104
Complete Your Online Session Evaluation
Give us your feedback and
you could win fabulous prizes.
Winners announced daily.
Receive 20 Cisco Daily Challenge
points for each session evaluation
you complete.
Complete your session evaluation
online now through either the mobile
app or internet kiosk stations.
Maximize your Cisco Live experience with your
free Cisco Live 365 account. Download session
PDFs, view sessions on-demand and participate in
live activities throughout the year. Click the Enter
Cisco Live 365 button in your Cisco Live portal to
log in.
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 105
Appendix

GETVPN Scalability and Troubleshooting Tools


Key Server Scalability
Platform Crypto Card Max Number of GM Time to register to KS

7200/7201 VAM2+ 2000 15 sec *

3845 AIM-VPN/SSL-3 1000 15 sec *

3825 AIM-VPN/SSL-3 500 15 sec

2851 AIM-VPN/SSL-2 200 15 sec

2821 AIM-VPN/SSL-2 100 15 sec

1841 AIM-VPN/SSL-1 50 15 sec

7200/PKI VAM2+ 1000 20 sec **

* GM registration was distributed over two KSs to reduce the registration time
** GM registration was distributed over four KSs to reduce the registration time

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
GM Performance Attributes
(No Features)

PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)
871 Anti-Replay 3150 28 <10
1.18
No Anti-Replay 3232 28 <5
1841-onboard Anti-Replay 3506 33 <20
1.07
No Anti-Replay 3766 35 <35
1841-aim/ssl Anti-Replay 8420 84 <10
0.68
No Anti-Replay 8472 84 <20
2821-onboard Anti-Replay 17152 50 <5
0.47
No Anti-Replay 17046 50 <1
2821-aim/ssl Anti-Replay 26010 190 <5
0.34
No Anti-Replay 25918 190 <5
2851-onboard Anti-Replay 17868 64 <5
0.33
No Anti-Replay 19175 65 <10
2851-aim/ssl Anti-Replay 27594 190 <1
0.25
No Anti-Replay 27668 190 <1

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
GM Performance Attributes
(No Features)

PPS Mbps Max IMIX Latency(ms) Avg 100 pps Latency (ms)
3825-onboard Anti-Replay 35,505 283 <1
0.64
No Anti-Replay 35,500 283 <5
3825-aim/ssl Anti-Replay 44,170 199 <1
0.66
No Anti-Replay 44,452 199 <5
3845-onboard Anti-Replay 46,028 284 <5
0.76
No Anti-Replay 46,028 283 <5
3845-aim/ssl Anti-Replay 54,020 200 <1
0.81
No Anti-Replay 53,996 200 <1
7200-g1vam2+ Anti-Replay 60,592 266 <5
0.69
No Anti-Replay 66,952 266 <5
7200-g2vam2+ Anti-Replay 121,952 283 <5
0.17
No Anti-Replay 120,890 283 <1
7200-g2/vsa Anti-Replay
TBD
No Anti-Replay 160,000 980 TBD
ASR1000/FP5G Anti-Replay 440,000
TBD
No Anti-Replay 470,000 1,890 TBD
ASR1000/FP10G Anti-Replay 976,000 4,200
0.19
No Anti-Replay 1,011,000 4,220 <0.270
ASR1000/FP20G Anti-Replay 2,655,000 TBD
0.001
No Anti-Replay 2,685,000 8,530 <0.015

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
GM Performance Attributes
(No Features)

Frame Size ASR 1004 (10Gig) 7200 VSA 3845 AIM- ISRG2 ISRG2 ISRG2
VPN/SSL-3 3945 Onboard 2951 Onboard 1941
Crypto Crypto Onboard
Crypto

1400 Byte 4759 Mbps 925 Mbps 200 Mbps 820 Mbps 268 Mbps 154
Mbps

IMIX (90 Bytes 61%, 2289 Mbps 780 Mbps 177 Mbps 261Mbps 160 Mbps 64Mbp
594 bytes 24%, 1418 15%) s

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
GETVPN Verification
Common KS Syslog Messages
Syslog Messages Explanation
COOP_CONFIG_MISMATCH The configuration between the primary key server and secondary key server are
mismatched.
COOP_KS_ELECTION The local key server has entered the election process in a group.
COOP_KS_REACH The reachability between the configured cooperative key servers is restored.

COOP_KS_TRANS_TO_PRI The local key server transitioned to a primary role from being a secondary server
in a group.
COOP_KS_UNAUTH An authorized remote server tried to contact the local key server in a group.
Could be considered a hostile event.
COOP_KS_UNREACH The reachability between the configured cooperative key servers is lost. Could
be considered a hostile event.
KS_GM_REVOKED During rekey protocol, an unauthorized member tried to join a group. Could be
considered a hostile event.
KS_SEND_MCAST_REKEY Sending multicast rekey.
KS_SEND_UNICAST_REKEY Sending unicast rekey.
KS_UNAUTHORIZED During GDOI registration protocol, an unauthorized member tried to join a group.
Could be considered a hostile event.
UNAUTHORIZED_IPADDR The registration request was dropped because the requesting device was not
authorized to join the group.

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
GETVPN Verification
Common GM Syslog Messages
Syslog Messages Explanation
GM_CLEAR_REGISTER The clear crypto gdoi command has been executed by the local
group member.
GM_CM_ATTACH A crypto map has been attached for the local group member.

GM_CM_DETACH A crypto map has been detached for the local group member.

GM_RE_REGISTER IPSec SA created for one group may have been expired or
cleared. Need to reregister to the key server
GM_RECV_REKEY Rekey received
GM_REGS_COMPL Registration complete
GM_REKEY_TRANS_2_MULTI Group member has transitioned from using a unicast rekey
mechanism to using a multicast mechanism.
GM_REKEY_TRANS_2_UNI Group member has transitioned from using a multicast rekey
mechanism to using a unicast mechanism.
PSEUDO_TIME_LARGE A group member has received a pseudotime with a value that is largely
different from its own pseudotime.
REPLAY_FAILED A group member or key server has failed an anti-replay check.

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
Packet marking Techniques

IP TOS byte copied from inner header to the encapsulating


delivery header by default
How to mark
PBR
MQC
Local ping
How to monitor
IP precedence accounting
ACL counters

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
ToS/Precedence/DSCP Reference Chart

Least
7 6 5 4 3 2 1 0 Significant
Bit
IP Precedence Priority
DSCP ToS Byte

ToS
Hex - Decimal IP Precedence DSCP Binary
E0 224 7 Network Control 56 CS7 11100000
C0 192 6 Internetwork Control 48 CS6 11000000
B8 184 5 Critical 46 EF 10111000
A0 160 40 CS5 10100000
88 136 4 Flash Override 34 AF41 10001000
80 128 32 CS4 10000000
68 104 3 Flash 26 AF31 01101000
60 96 24 CS3 01100000
48 72 2 Immediate 18 AF21 01001000
40 64 16 CS2 01000000
20 32 1 Priority 8 CS1 00100000
00 0 0 Routine 0 Dflt 00000000
BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
Packet marking - Examples
PBR
interface Ethernet1/0
ip policy route-map mark
!
access-list 150 permit ip host 172.16.1.2 host 172.16.254.2
!
route-map mark permit 10
match ip address 150
set ip precedence flash-override

MQC IP flow in question marked with


precedence 4
class-map match-all my_flow
match access-group 150
!
policy-map marking
class my_flow
set ip precedence 4
!
interface Ethernet1/0
service-policy input marking

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
Packet marking - Examples

Router Ping
Router#ping ip
Target IP address: 172.16.254.2
Repeat count [5]: 100
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 128
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 100, 100-byte ICMPEchos to 172.16.254.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
Packet marking - Monitoring
IP Precedence Accounting
interface Ethernet0/0
ip address 192.168.1.2 255.255.255.0
ip accounting precedence input

middle_router#show interface precedence


Ethernet0/0
Input
Precedence 4: 100 packets, 17400 bytes

Interface ACL
middle_router#sh access-list 144
Extended IP access list 144
10 permit ip any any precedence routine
20 permit ip any any precedence priority
30 permit ip any any precedence immediate
40 permit ip any any precedence flash
50 permit ip any any precedence flash-override (100 matches)
60 permit ip any any precedence critical
70 permit ip any any precedence internet (1 match)
80 permit ip any any precedence network

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
Using Packet Captures for Data Plane Issues
Packet captures can provide detailed packet information at the bits/bytes level
The new packet capture infrastructure introduced in 12.4(20)T makes this easy
to do
Ability to capture IPv4 and IPv6 packets in the CEF path
Configurable capture buffer and capture point parameters
Extensible output filtering and export capabilities
Support for various WAN encapsulation types

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
Using IOS Embedded Packet Captures
Key Configuration Steps
Create the capture buffer and capture point
Associate the capture point to the buffer
Start/stop the capture

Router#monitor capture buffer test-buffer


Router#monitor capture buffer test-buffer filter access-list 120
Filter Association succeeded
Router#
Router#monitor capture point ipcef test-capture serial 2/0 both
*Mar 26 20:33:10.896: %BUFCAP-6-CREATE: Capture Point test-capture created.
Router#monitor capture point associate test-capture test-buffer
Router#monitor capture point start test-capture
*Mar 26 20:34:03.108: %BUFCAP-6-ENABLE: Capture Point test-capture enabled.
Router#
Router#monitor capture point stop test-capture
*Mar 26 20:34:21.636: %BUFCAP-6-DISABLE: Capture Point test-capture disabled.

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
Using IOS Embedded Packet Captures
Now we have the packets captured, whats next?
Dump the packet on the router itself
Router# show monitor capture buffer test-buffer dump
15:34:07.228 EST Mar 26 2009 : IPv4 LES CEF : Se2/0 None

05CECE30: 0F00080045C0002C ....E@.,


05CECE40: 6D170000FE0649DD 02010102 01010114 m...~.I]........
05CECE50: 0017A3530FB6B9523EF1499C 60121020 ..#S.69R>qI.`..
05CECE60: 917A0000 02040218 00 .z.......

Dump the packet on the router itself


Or export it out and analyze it in Wireshark
Router# monitor capture buffer test-buffer export?
ftp: Location to dump buffer
http: Location to dump buffer
https: Location to dump buffer
rcp: Location to dump buffer
scp: Location to dump buffer
tftp: Location to dump buffer

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public
Use EEM and EPC to catch Packet Corruption
Peer1
event manager applet detect_bad_packet
event syslog pattern "RECVD_PKT_MAC_ERR"
action 1.0 cli command "enable"
action 2.0 cli command "monitor capture point stop test"
action 3.0 syslog msg "Packet corruption detected and capture stopped!"
action 4.0 snmp-trap intdata1 123456 strdata ""

Peer2

event manager applet detect_bad_packet


event snmp-notification oid 1.3.6.1.4.1.9.10.91.1.2.3.1.9. oid-val "123456" op
eq src-ip-address 20.1.1.1
action 1.0 cli command "enable"
action 2.0 cli command "monitor capture point stop test"
action 3.0 syslog msg "Packet corruption detected and capture stopped!"

BRKSEC-3051 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public

Оценить