JNCIE-SEC Traceoptions & IPSEC

troubleshooting
In IPSEC topic, I am continuing with traceoptions and troubleshooting section. In this post, I will
try to explain how I troubleshoot IPSEC VPNs mostly initial setup.
IPsec VPNs

Implementation of IPsec VPNs

Multipoint tunnels

Policy and route-based VPNs

Traceoptions

Dual and backup tunnels

On-demand tunnels

DRP over a tunnel

Dynamic VPNs

Certificate-based VPNs

PKI

Interoperability with 3rd party devices

NAT

Implementation of NAT

Source NAT

Destination NAT

Static NAT

Implementation of NAT with IPSec

Overlapping IPs between sites

One of the challenging parts of JNCIE-SEC must be the troubleshooting part for which I need to
understand under what sort of problems what type of error logs are generated. Because of this, I
enabled IKE traceoptions and simulated several type of possible problems and observed the error
logs.
But first lets see how a successful IKE Phase 1 and IKE Phase 2 log looks like;
PS: All errors below are between ike peers 192.168.179.2 and 212.45.64.2
IKE & IPSEC SUCCESSFUL LOG
Phase 1
[Aug
22 20:40:14]ike_calc_mac: Star
[Aug 22 20:40:14]ike_st_i_cert:
[Aug 22 20:40:14]ike_st_i_priva

1
2
3
4

[Aug 22 20:40:14]ike_calc_mac: Start, initiator = true, local = false

[Aug 22 20:40:14]ike_st_i_cert: Start
[Aug 22 20:40:14]ike_st_i_private: Start
[Aug 22 20:40:14]ike_st_o_wait_done: Marking for waiting for done

[Aug 22 20:40:14]ike_st_o_all_done: MESSAGE: Phase 1 { 0xe4d65d2e a7bf1c17 0x498aaa01 01d0dd21 } / 00000000, version = 1.0, xchg = Identity protect, auth_method =
Pre shared keys, Initiator, cipher = 3des-cbc, hash = sha1, prf = hmac-s
ha1
[Aug 22 20:40:14]192.168.179.2:500 (Initiator) <-> 212.45.64.2:500 { e4d65d2e a7bf1c17 5 498aaa01 01d0dd21 [-1] / 0x00000000 } IP; MESSAGE: Phase 1 version = 1.0, auth_method
6 = Pre shared keys, cipher = 3des-cbc, hash = sha1, prf = hmac7 sha
8 [Aug 22 20:40:14]ike_send_notify: Connected, SA = { e4d65d2e a7bf1c17 - 498aaa01
9 01d0dd21}, nego = -1
10 [Aug 22 20:40:14]iked_pm_ike_sa_done: local:192.168.179.2, remote:212.45.64.2 IKEv1
11 [Aug 22 20:40:14]IKE negotiation done for local:192.168.179.2, remote:212.45.64.2 IKEv1
12 with status: Error ok
13 [Aug 22 20:40:14]Added (spi=0xaebf2827, protocol=0) entry to the spi table
14 [Aug 22 20:40:14]Added (spi=0x3037b766, protocol=0) entry to the spi table
15 [Aug 22 20:40:14]ssh_ike_connect_ipsec: Start, remote_name = :500, flags = 00000000
16 [Aug 22 20:40:14]ike_sa_find_ip_port: Remote = all:500, Found SA = { e4d65d2e a7bf1c17
17 - 498aaa01 01d0dd21}
18 [Aug 22 20:40:14]ike_alloc_negotiation: Start, SA = { e4d65d2e a7bf1c17 - 498aaa01
19 01d0dd21}
20 [Aug 22 20:40:14]ssh_ike_connect_ipsec: SA = { e4d65d2e a7bf1c17 - 498aaa01
21 01d0dd21}, nego = 0
22 [Aug 22 20:40:14]ike_init_qm_negotiation: Start, initiator = 1, message_id = 5aa9f0f2
23 [Aug 22 20:40:14]ike_st_o_qm_hash_1: Start
24 [Aug 22 20:40:14]ike_st_o_qm_sa_proposals: Start
25 [Aug 22 20:40:14]ike_st_o_qm_nonce: Start
26 [Aug 22 20:40:14]ike_policy_reply_qm_nonce_data_len: Start
27 [Aug 22 20:40:14]ike_st_o_qm_optional_ke: Start
28 [Aug 22 20:40:14]ike_st_o_qm_optional_ids: Start
[Aug 22 20:40:14]ike_st_qm_optional_id: Start
[Aug 22 20:40:14]ike_st_qm_optional_id: Start
[Aug 22 20:40:14]ike_st_o_private: Start
[Aug 22 20:40:14]Construction NHTB payload for local:192.168.179.2, remote:212.45.64.2
IKEv1 P1 SA index 8160872 sa-cfg vpn-hub
You can see the IKE negotiation done log in here.
Phase 2
[Aug
22 20:40:14]ike_policy_reply_p
[Aug 22 20:40:14]ike_st_o_encr
[Aug 22 20:40:14]<none>:500 (I

1 [Aug 22 20:40:14]ike_policy_reply_private_payload_out: Start

[Aug 22 20:40:14]ike_st_o_encrypt: Marking encryption for packet

[Aug 22 20:40:14]<none>:500 (Initiator) <-> 212.45.64.2:500 { e4d65d2e a7bf1c17 498aaa01 01d0dd21 [0] / 0x5aa9f0f2 } QM; MESSAGE: Phase 2 connection succeeded, No
PFS, group = 0
[Aug 22 20:40:14]ike_qm_call_callback: MESSAGE: Phase 2 connection succeeded, No
PFS, group = 0
2
[Aug 22 20:40:14]<none>:500 (Initiator) <-> 212.45.64.2:500 { e4d65d2e a7bf1c17 3
498aaa01 01d0dd21 [0] / 0x5aa9f0f2 } QM; MESSAGE: SA[0][0] = ESP 3des, life = 0
4
kB/3600 sec, group = 0, tunnel, hmac-sha1-96, Extended seq not used, key le
5
n=
6
[Aug 22 20:40:14]ike_qm_call_callback: MESSAGE: SA[0][0] = ESP 3des, life = 0 kB/3600
7
sec, group = 0, tunnel, hmac-sha1-96, Extended seq not used, key len = 0, key rounds = 0
8
[Aug 22 20:40:14]iked_pm_ipsec_sa_install: local:192.168.179.2,
9
remote:212.45.64.2 IKEv1 for SA-CFG vpn-hub
10
[Aug 22 20:40:14]Added (spi=0xaebf2827, protocol=ESP dst=192.168.179.2) entry to the
11
peer hash table
12
[Aug 22 20:40:14]Added (spi=0xdfa0760c, protocol=ESP dst=212.45.64.2) entry to the peer
13
hash table
14
[Aug 22 20:40:14]Hardlife timer started for inbound vpn-hub with 3600 seconds/0 kilobytes
15
[Aug 22 20:40:14]Softlife timer started for inbound vpn-hub with 2978 seconds/0 kilobytes
16
[Aug 22 20:40:14]In iked_ipsec_sa_pair_add Adding GENCFG msg with key; Tunnel =
17
131073;SPI-In = 0xaebf2827
[Aug 22 20:40:14]Added dependency on SA config blob with tunnelid = 131073
[Aug 22 20:40:14]Successfully added ipsec SA PAIR
[Aug 22 20:40:14]ike_st_o_qm_wait_done: Marking for waiting for done
[Aug 22 20:40:14]ike_encode_packet: Start, SA = { 0xe4d65d2e a7bf1c17 - 498aaa01
01d0dd21 } / 5aa9f0f2, nego = 0
You can see QM; MESSAGE: Phase 2 connection succeeded message.
For this specific connection here is the CLI outputs;
root@J23-London> show secu
Index State Initiator cookie Re
8160872 UP e4d65d2ea7bf1c

1
2
3
4
5
6
7
8

root@J23-London> show security ike sa

Index State Initiator cookie Responder cookie Mode
Remote Address
8160872 UP e4d65d2ea7bf1c17 498aaa0101d0dd21 Main
212.45.64.2
root@J23-London> show security ipsec sa
Total active tunnels: 1
ID Algorithm
SPI
Life:sec/kb Mon lsys Port Gateway
<131073 ESP:3des/sha1 aebf2827 3469/ unlim - root 500 212.45.64.2

9 >131073 ESP:3des/sha1 dfa0760c 3469/ unlim - root 500 212.45.64.2

ERROR 1: IKEv1 Error : Invalid payload type
If your pre-shared keys arent matching you will get a similar error log like below.
[Aug
22 20:49:08]<none>:500 (Resp
287c0904 ed0108cf - 1dd10036
Notification data has attribute lis

1 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.45.64.2:500 { 287c0904 ed0108cf 2 1dd10036 f9452830 [0] / 0x28e022de } Info; Notification data has attribute list
3 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.45.64.2:500 { 287c0904 ed0108cf 4 1dd10036 f9452830 [0] / 0x28e022de } Info; Notify message version = 1
5 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.45.64.2:500 { 287c0904 ed0108cf 6 1dd10036 f9452830 [0] / 0x28e022de } Info; Offending payload type = 145
7 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.45.64.2:500 { 287c0904 ed0108cf 8 1dd10036 f9452830 [0] / 0x28e022de } Info; Offending payload data offset = 0
9 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.45.64.2:500 { 287c0904 ed0108cf 10 1dd10036 f9452830 [0] / 0x28e022de } Info; Error text = Incorrect pre-shared key (Invalid
11 next payload value)
12 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.45.64.2:500 { 287c0904 ed0108cf 13 1dd10036 f9452830 [0] / 0x28e022de } Info; Offending message id = 0x00000000
14 [Aug 22 20:49:08]<none>:500 (Responder) <-> 212.45.64.2:500 { 287c0904 ed0108cf 15 1dd10036 f9452830 [0] / 0x28e022de } Info; Received notify err = Invalid payload type (1)
16 to isakmp sa, delete it
17 [Aug 22 20:49:08]ike_st_i_private: Start
18 [Aug 22 20:49:08]ike_send_notify: Connected, SA = { 287c0904 ed0108cf - 1dd10036
19 f9452830}, nego = 0
20 [Aug 22 20:49:08]ike_delete_negotiation: Start, SA = { 287c0904 ed0108cf - 1dd10036
21 f9452830}, nego = 0
22 [Aug 22 20:49:08]ike_free_negotiation_info: Start, nego = 0
23 [Aug 22 20:49:08]ike_free_negotiation: Start, nego = 0
24 [Aug 22 20:49:08]ike_remove_callback: Start, delete SA = { 287c0904 ed0108cf - 1dd10036
25 f9452830}, nego = -1
26 [Aug 22 20:49:08]192.168.179.2:500 (Initiator) <-> 212.45.64.2:500 { 287c0904 ed0108cf 1dd10036 f9452830 [-1] / 0x00000000 } IP; Connection got error = 1, calling callback
[Aug 22 20:49:08]ike_delete_negotiation: Start, SA = { 287c0904 ed0108cf - 1dd10036
f9452830}, nego = -1
[Aug 22 20:49:08]ssh_ike_tunnel_table_entry_delete: Deleting tunnel_id: 0 from IKE tunnel
table
[Aug 22 20:49:08]ssh_ike_tunnel_table_entry_delete: The tunnel id: 0 doesn't exist in IKE
tunnel table
[Aug 22 20:49:08]ike_sa_delete: Start, SA = { 287c0904 ed0108cf - 1dd10036 f9452830 }
[Aug 22 20:49:08]ike_free_negotiation_isakmp: Start, nego = -1

[Aug 22 20:49:08]ike_free_negotiation: Start, nego = -1

[Aug 22 20:49:08]ike_free_id_payload: Start, id type = 1
[Aug 22 20:49:08]ike_free_sa: Start
[Aug 22 20:49:08]IKE negotiation fail for local:192.168.179.2, remote:212.45.64.2 IKEv1
with status: Invalid syntax
[Aug 22 20:49:08] IKEv1 Error : Invalid payload type
[Aug 22 20:49:08]IPSec Rekey for SPI 0x0 failed
[Aug 22 20:49:08]IPSec SA done callback called for sa-cfg vpn-hub local:192.168.179.2,
remote:212.45.64.2 IKEv1 with status Invalid syntax
Error message IKEv1 Error: Invalid payload type is a likely indication of a pre-shared key
mismatch. You can also see Error text = Incorrect pe-shared-key
Error 2: IKEv1 Error : No proposal chosen
You will get the following error if one of the followings mismatches in your IKE config;

dh-group

authentication algorithm

encryption algorithm

WARNING!!!: In addition to these mismatches, you will get the same error under the following
conditions

if you forget to set bind-interface st0.0 under your vpn configuration,

if st0.0 interface isnt created with family inet and/or assigned to a security zone

[Aug
22 20:53:10]ike_st_i_n: Start, d
proposal chosen (14), spi[0..16
800c0001 00060022 ...

1 [Aug 22 20:53:10]ike_st_i_n: Start, doi = 1, protocol = 1, code = No proposal chosen (14),

2 spi[0..16] = f39ace76 bde7864a ..., data[0..46] = 800c0001 00060022 ...
3 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.45.64.2:500 { f39ace76 bde7864a 4 0d31547b 7e819258 [0] / 0xf638af05 } Info; Notification data has attribute list
5 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.45.64.2:500 { f39ace76 bde7864a 6 0d31547b 7e819258 [0] / 0xf638af05 } Info; Notify message version = 1
7 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.45.64.2:500 { f39ace76 bde7864a 8 0d31547b 7e819258 [0] / 0xf638af05 } Info; Error text = Could not find acceptable proposal
9 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.45.64.2:500 { f39ace76 bde7864a 10 0d31547b 7e819258 [0] / 0xf638af05 } Info; Offending message id = 0x00000000
11 [Aug 22 20:53:10]<none>:500 (Responder) <-> 212.45.64.2:500 { f39ace76 bde7864a -

0d31547b 7e819258 [0] / 0xf638af05 } Info; Received notify err = No proposal chosen (14)
to isakmp sa, delete it
[Aug 22 20:53:10]ike_st_i_private: Start
[Aug 22 20:53:10]ike_send_notify: Connected, SA = { f39ace76 bde7864a - 0d31547b
7e819258}, nego = 0
[Aug 22 20:53:10]ike_delete_negotiation: Start, SA = { f39ace76 bde7864a - 0d31547b
7e819258}, nego = 0
12
[Aug 22 20:53:10]ike_free_negotiation_info: Start, nego = 0
13
[Aug 22 20:53:10]ike_free_negotiation: Start, nego = 0
14
[Aug 22 20:53:10]ike_remove_callback: Start, delete SA = { f39ace76 bde7864a - 0d31547b
15
7e819258}, nego = -1
16
[Aug 22 20:53:10]192.168.179.2:500 (Initiator) <-> 212.45.64.2:500 { f39ace76 bde7864a 17
0d31547b 7e819258 [-1] / 0x00000000 } IP; Connection got error = 14, calling callback
18
[Aug 22 20:53:10]ikev2_fb_v1_encr_id_to_v2_id: Unknown IKE encryption identifier -1
19
[Aug 22 20:53:10]ikev2_fb_v1_hash_id_to_v2_prf_id: Unknown IKE hash alg identifier -1
20
[Aug 22 20:53:10]ikev2_fb_v1_hash_id_to_v2_integ_id: Unknown IKE hash alg identifier -1
[Aug 22 20:53:10]IKE negotiation fail for local:192.168.179.2, remote:212.45.64.2 IKEv1
with status: No proposal chosen
[Aug 22 20:53:10] IKEv1 Error : No proposal chosen
[Aug 22 20:53:10]IPSec Rekey for SPI 0x0 failed
[Aug 22 20:53:10]IPSec SA done callback called for sa-cfg vpn-hub local:192.168.179.2,
remote:212.45.64.2 IKEv1 with status No proposal chosen
ERROR 3: IKEv1 Error : Timeout
If IKE port 500 isnt reachable at the peer, you will get an error like this. Another problem you
might encounter is that for example, you forget to enable IKE service in a zone only in one peer
(e.g Peer B) but Peer A is still allowing IKE. This means peer A cant be the initiator but only
responder. Because A cant connect to IKE port but B can.
[Aug
22 20:59:08]KMD_INTERNAL_E
received
[Aug 22 20:59:13]iked_spu_ha_

1 [Aug 22 20:59:08]KMD_INTERNAL_ERROR: iked_ifstate_eoc_handler: EOC msg

2 received
3 [Aug 22 20:59:13]iked_spu_ha_ipc_get_server_addr, server tnp addr (standalone): 0x1, ISSU
4 pending=no
5 [Aug 22 20:59:24]iked_pm_ike_spd_notify_request: Sending Initial contact
6 [Aug 22 20:59:24]ssh_ike_connect: Start, remote_name = 212.45.64.2:500, xchg = 2, flags =
7 00090000
8 [Aug 22 20:59:24]ike_sa_allocate: Start, SA = { e6ed730d 487d645f - 00000000 00000000 }
9 [Aug 22 20:59:24]ike_init_isakmp_sa: Start, remote = 212.45.64.2:500, initiator = 1
10 [Aug 22 20:59:24]ssh_ike_connect: SA = { e6ed730d 487d645f - 00000000 00000000}, nego
11 = -1

[Aug 22 20:59:24]ike_st_o_sa_proposal: Start

[Aug 22 20:59:24]ike_policy_reply_isakmp_vendor_ids: Start
[Aug 22 20:59:24]ike_st_o_private: Start
[Aug 22 20:59:24]ike_policy_reply_private_payload_out: Start
[Aug 22 20:59:24]ike_encode_packet: Start, SA = { 0xe6ed730d 487d645f - 00000000
00000000 } / 00000000, nego = -1
[Aug 22 20:59:24]ike_send_packet: Start, send SA = { e6ed730d 487d645f - 00000000
00000000}, nego = -1, dst = 212.45.64.2:500, routing table id = 0
12
[Aug 22 20:59:34]ike_retransmit_callback: Start, retransmit SA = { e6ed730d 487d645f 13
00000000 00000000}, nego = -1
14
[Aug 22 20:59:34]ike_send_packet: Start, retransmit previous packet SA = { e6ed730d
15
487d645f - 00000000 00000000}, nego = -1, dst = 212.45.64.2:500 routing table id = 0
16
[Aug 22 20:59:44]ike_retransmit_callback: Start, retransmit SA = { e6ed730d 487d645f 17
00000000 00000000}, nego = -1
18
[Aug 22 20:59:44]ike_send_packet: Start, retransmit previous packet SA = { e6ed730d
19
487d645f - 00000000 00000000}, nego = -1, dst = 212.45.64.2:500 routing table id = 0
20
[Aug 22 20:59:54]P1 SA 2299946 timer expiry. ref cnt 2, timer reason Force delete timer
21
expired (1), flags 0x0.
22
[Aug 22 20:59:54]iked_pm_ike_sa_delete_done_cb: For p1 sa index 2299946, ref cnt 2,
23
status: Error ok
24
[Aug 22 20:59:54]ike_remove_callback: Start, delete SA = { e6ed730d 487d645f - 00000000
25
00000000}, nego = -1
26
[Aug 22 20:59:54]192.168.179.2:500 (Initiator) <-> 212.45.64.2:500 { e6ed730d 487d645f 27
00000000 00000000 [-1] / 0x00000000 } IP; Connection timed out or error, calling callback
28
[Aug 22 20:59:54]ikev2_fb_v1_encr_id_to_v2_id: Unknown IKE encryption identifier -1
[Aug 22 20:59:54]ikev2_fb_v1_hash_id_to_v2_prf_id: Unknown IKE hash alg identifier -1
[Aug 22 20:59:54]ikev2_fb_v1_hash_id_to_v2_integ_id: Unknown IKE hash alg identifier -1
[Aug 22 20:59:54]iked_pm_ike_sa_done: UNUSABLE p1_sa 2299946
[Aug 22 20:59:54] IKEv1 Error : Timeout
[Aug 22 20:59:54]IPSec Rekey for SPI 0x0 failed
[Aug 22 20:59:54]IPSec SA done callback called for sa-cfg vpn-hub local:192.168.179.2,
remote:212.45.64.2 IKEv1 with status Timed out
ERROR 4: IKEv1 Error : No proposal chosen
This is the same error like ERROR 2 but it is actually caused by IPSEC proposals not IKE. Thus
if one of the following two mismatches, you will get this error.

Authentication algorithm

Encryption algorithm

Note: I had thought that ipsec lifetime is also something that has to match but my tests showed a
different result. As far as I can see peers agree on the lowest lifetime configured i.e if peer A has
3600secs and peer B has 7200secs, they agree on 3600secs.

FLOW Troubleshooting
So far I have done IKE troubleshooting. Now I will do some flow troubleshooting.
I enable traceoptions for the traffic that I am going to generate.
[edit]
root@J23-London# show secu
traceoptions {
file ipsec-traf.log size 5m;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

[edit]
root@J23-London# show security flow
traceoptions {
file ipsec-traf.log size 5m;
flag basic-datapath;
packet-filter outgoing-filter {
protocol icmp;
source-prefix 55.55.55.1/32;
destination-prefix 212.45.63.1/32;
}
packet-filter incoming-filter {
protocol icmp;
source-prefix 212.45.63.1/32;
destination-prefix 55.55.55.1/32;
}
packet-filter outgoing-esp {
protocol esp;
source-prefix 192.168.179.2/32;
destination-prefix 212.45.64.2/32;
}
}
First two packet filters will show us clear text packets but outgoing-esp is for the encrypted
packets
Lets send 1 ICMP packet with 1000 bytes (1008 bytes with ICMP header)
root@J23-London> ping 212.45
PING 212.45.63.1 (212.45.63.1)
1008 bytes from 212.45.63.1: ic

1 root@J23-London> ping 212.45.63.1 source 55.55.55.1 count 1 size 1000

2 PING 212.45.63.1 (212.45.63.1): 1000 data bytes
3 1008 bytes from 212.45.63.1: icmp_seq=0 ttl=64 time=46.204 ms

4
5 --- 212.45.63.1 ping statistics --6 1 packets transmitted, 1 packets received, 0% packet loss
7 round-trip min/avg/max/stddev = 46.204/46.204/46.204/0.000 ms
Now examine the file ipsec-traf.log
outgoing-filter match
Aug
22 20:01:06
20:01:06.574883:CID-0:RT:<55.5
matched filter outgoing-filter:

1 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:<55.55.55.1/0->212.45.63.1/23813;1> matched

2 filter outgoing-filter:
3 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:packet [1028] ipid = 3812, @0x4d10d6c1
4 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type
5 0, common flag 0x0, mbuf 0x4d10d480, rtbl_idx = 0
6 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: in_ifp <junos-host:.local..0>
7 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_process_pkt_exception: setting rtt in lpak
8 to 0x5d22ca70
9 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Using vr id from pfe_tag with value= 0
10 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Changing lpak->in_ifp from:.local..0 ->
11 to:.local..0
12 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Over-riding lpak->vsys with 0
13 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: .local..0:55.55.55.1->212.45.63.1, icmp, (8/0)
14 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: find flow: table 0x552e9c90, hash
15 28352(0xffff), sa 55.55.55.1, da 212.45.63.1, sp 0, dp 23813, proto 1, tok 2
16 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: no session found, start first path. in_tunnel 17 0x0, from_cp_flag - 0
18 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: flow_first_create_session
19 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:(flow_first_create_session) usp_tagged set
20 session as mng session
21 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: flow_first_in_dst_nat: in <.local..0>, out
22 <N/A> dst_adr 212.45.63.1, sp 0, dp 23813
23 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: chose interface .local..0 as incoming nat if.
24 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_rule_dst_xlate: packet 55.55.55.125 >212.45.63.1 nsp2 0.0.0.0->212.45.63.1.
26 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_routing: vr_id 0, call
27 flow_route_lookup(): src_ip 55.55.55.1, x_dst_ip 212.45.63.1, in ifp .local..0, out ifp N/A sp
28 0, dp 2
29 3813, ip_proto 1, tos 0
30 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Doing DESTINATION addr route-lookup
31 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: routed (x_dst_ip 212.45.63.1) from junos-host

32 (.local..0 in 0) to st0.0, Next-hop: 212.45.63.1

33 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_policy_search: policy search from
34 zone junos-host-> zone vpn (0x0,0x5d05,0x5d05)
35 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Policy lkup: vsys 0 zone(2:junos-host) ->
36 zone(7:vpn) scope:0
37 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:
55.55.55.1/2048 -> 212.45.63.1/38837
38 proto 1
39 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: app 0, timeout 60s, curr ageout 60s
40 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: permitted by policy self-traffic-policy(1)
41 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: packet passed, Permitted by policy.
42 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_src_xlate: nat_src_xlated: False,
43 nat_src_xlate_failed: False
44 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_src_xlate: src nat returns status: 0,
45 rule/pool id: 0/0, pst_nat: False.
46 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: dip id = 0/0, 55.55.55.1/0->55.55.55.1/0
47 protocol 0
48 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: Found tunnel for if (non-vpn or vpn without
49 nhtb) st0.0
50 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_get_tun_info: tunnel out
51 0x577cf0ec, tun id 131073
52 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_get_out_ifp: tunnel out 0x577cf0ec,
53 tun id 131073
54 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: choose interface ge-0/0/0.0 as outgoing phy if
55 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:is_loop_pak: No loop: on ifp: st0.0, addr:
56 212.45.63.1, rtt_idx:0
57 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf : Alloc sess plugin info for session 8
58 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:[JSF]Normal interest check. regd plugins 12,
59 enabled impl mask 0x0
60 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 2, svc_req 0x0, impl
61 mask 0x0. rc 4
62 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 3, svc_req 0x0, impl
63 mask 0x0. rc 4
64 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 12, svc_req 0x0, impl
65 mask 0x0. rc 4
66 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: Error : parameter wrong natp 0x577cfcc8,
67 plugin_id 0
68 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 15, svc_req 0x0, impl
69 mask 0x0. rc 4
70 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:+++++++++++jsf_test_plugin_data_evh: 3
71 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 16, svc_req 0x0, impl
72 mask 0x0. rc 4
73 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 22, svc_req 0x0, impl
mask 0x0. rc 4
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf int check: plugin id 27, svc_req 0x0, impl
mask 0x0. rc 2
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:[JSF]Plugins(0x0, count 0) enabled for session

= 34359738368, impli mask(0x0), post_nat cnt 0 svc req(0x0)

Aug 22 20:01:06 20:01:06.574883:CID-0:RT:-jsf : no plugin interested for session 8, free
sess plugin info
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_service_lookup(): natp(0x577cfcc8):
app_id, 0(0).
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: service lookup identified service 0.
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: flow_first_final_check: in <.local..0>, out
<ge-0/0/0.0>
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_first_complete_session, pak_ptr:
0xbf97d578, nsp: 0x577cfcc8, in_tunnel: 0x0
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:construct v4 vector for nsp2
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: existing vector list 0x204-0x50a92108.
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: Session (id:8) created for first pak 204
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: flow_first_install_session======>
0x577cfcc8
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: nsp 0x577cfcc8, nsp2 0x577cfd48
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: make_nsp_ready_no_resolve()
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: route lookup: dest-ip 55.55.55.1 orig ifp
.local..0 output_ifp .local..0 orig-zone 2 out-zone 2 vsd 0
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: route to 55.55.55.1
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:avt_get_config_by_lsys_id: Not supported on
low memory platforms.
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:no need update ha
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:Installing c2s NP session wing
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: flow got session.
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: flow session id 8
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: vector bits 0x204 vector 0x50a92108
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:ttl vector, out_tunnel = 0x577cf0ec
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:pre-frag not needed: ipsize: 1028, mtu: 1438,
nsp2->pmtu: 1438
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: encap vector
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: going into tunnel 131073
(nsp_tunnel=0x577cf0ec).
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: flow_encrypt: tun 0x577cf0ec, type 1
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:mbuf 0x4d10d480, exit nh 0x30010
Aug 22 20:01:06 20:01:06.574883:CID-0:RT:flow_process_pkt_exception: Freeing lpak
0xbf97d578 associated with mbuf 0x4d10d480
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)
What happens here is ;

packet with size 1028 (extra 20 byte from IP header) with identification number 3812
matches: packet [1028] ipid = 3812

A new session is created flow_first_create_session

self-traffic-policy allows the traffic as it is locally generated

Tunnel id is identified for the traffic flow_first_get_tun_info: tunnel out 0x577cf0ec, tun
id 131073

Physical outgoing interface is chosen: choose interface ge-0/0/0.0 as outgoing phy if

and finally packet is encrypted flow_encrypt: tun 0x577cf0ec, type 1

outgoing-esp filter match

Aug
22 20:01:06
20:01:06.574883:CID-0:RT:<192
matched filter outgoing-esp:

Aug 22 20:01:06 20:01:06.574883:CID-0:RT:<192.168.179.2/0->212.45.64.2/0;50> matched

1 filter outgoing-esp:
2 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:packet [1080] ipid = 405, @0x4d10d69d
3 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type 2,
4 common flag 0x0, mbuf 0x4d10d480, rtbl_idx = 0
5 Aug 22 20:01:06 20:01:06.574883:CID-0:RT: post_encap: nsp_tunnel 0x577cf0ec. is_valid 1
6 Aug 22 20:01:06 20:01:06.574883:CID-0:RT:mbuf 0x4d10d480, exit nh 0xa0010
Aug 22 20:01:06 20:01:06.574883:CID-0:RT: ----- flow_process_pkt rc 0x0 (fp rc 0)
In this filter we can see that:
Packet is in the tunnel and grew in size to 1080 bytes (i.e total length of the new IP packet with
ESP header and encryption) and with outside ip id 405
incoming encrypted traffic
I have seen that though I havent configured returned esp traffic filter, outgoing-esp filter caught
this traffic.
Aug
22 21:02:44
21:02:44.579966:CID-0:RT:<212
matched filter outgoing-esp:

1 Aug 22 21:02:44 21:02:44.579966:CID-0:RT:<212.45.64.2/0->192.168.179.2/0;50> matched

2 filter outgoing-esp:
3 Aug 22 21:02:44 21:02:44.579966:CID-0:RT:packet [1080] ipid = 4, @0x49fa83ce
4 Aug 22 21:02:44 21:02:44.579966:CID-0:RT:---- flow_process_pkt: (thd 1): flow_ctxt type
5 15, common flag 0x0, mbuf 0x49fa8180, rtbl_idx = 0
6 Aug 22 21:02:44 21:02:44.579966:CID-0:RT: flow process pak fast ifl 70 in_ifp ge-0/0/0.0

Aug 22 21:02:44 21:02:44.579966:CID-0:RT: ge-0/0/0.0:212.45.64.2->192.168.179.2, 50

Aug 22 21:02:44 21:02:44.579966:CID-0:RT: find flow: table 0x552e9c90, hash
7
44389(0xffff), sa 212.45.64.2, da 192.168.179.2, sp 58759, dp 3018, proto 50, tok 8
8
Aug 22 21:02:44 21:02:44.579966:CID-0:RT: flow got session
9
Aug 22 21:02:44 21:02:44.579966:CID-0:RT: flow session id 1
Aug 22 21:02:44 21:02:44.579966:CID-0:RT: flow_decrypt: tun 0x577cf0ec(flag 0x82), iif 70

Return traffic enters from ge-0/0/0.0 interface : ge-0/0/0.0:212.45.64.2->192.168.179.2,

50

So called source port: 58759 and destination port: 3018

It hits the flow session with id 1 which is a one direction ESP session
root@J23-London> show secu
Session ID: 1, Policy name: N/A,
In: 212.45.64.2/58759 --> 192.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

root@J23-London> show security flow session protocol esp

Session ID: 1, Policy name: N/A, Timeout: N/A, Valid
In: 212.45.64.2/58759 --> 192.168.179.2/3018;esp, If: ge-0/0/0.0, Pkts: 0, Bytes: 0
Session ID: 2, Policy name: N/A, Timeout: N/A, Valid
In: 212.45.64.2/0 --> 192.168.179.2/0;esp, If: ge-0/0/0.0, Pkts: 0, Bytes: 0
Total sessions: 2
root@J23-London> show security flow session session-identifier 1
Session ID: 1, Status: Normal
Flag: 0x10000
Policy name: N/A
Source NAT pool: Null
Maximum timeout: N/A, Current timeout: N/A
Session State: Valid
Start time: 14, Duration: 7792
In: 212.45.64.2/58759 --> 192.168.179.2/3018;esp,
Interface: ge-0/0/0.0,
Session token: 0x8, Flag: 0x100621
Route: 0xa0010, Gateway: 192.168.179.1, Tunnel: 0
Port sequence: 0, FIN sequence: 0,
FIN state: 0,
Pkts: 0, Bytes: 0

I hope to have covered various scenarios in this post related to traceoptions and
troubleshooting of IPSEC VPN sessions. In the future posts, I will do some
troubleshooting if required.
If you have any other error you have received which isnt covered here, please do share.
DISCLAIMER: Views expressed in this blog are my own and do not necessarily reflect
those of Juniper Networks