VXLAN TROUBLESHOOTING AUTOMATION

submitted in partial fulfillment of the requirements

for the award of the degree
of

Master of Computer Applications

(2014-2017)

By:
Meghna Acharjee
02204092014

Under the Guidance of:

Ms. Ankita Singh
Assistant Professor

DEPARTMENT OF INFORMATION TECHNOLOGY

INDIRA GANDHI DELHI TECHNICAL UNIVERSITY FOR WOMEN
KASHMERE GATE, DELHI- 110006
 CERTIFICATE

I, Meghna Acharjee (02204092014), certify that the MCA Dissertation Project Report
entitled VxLAN Troubleshooting Automation is done by me and it is an authentic work
carried out by me at Cisco Systems Pvt. Ltd.. The matter embodied in this project work has
not been submitted earlier for the award of any degree or diploma to the best of my knowledge
and belief.

MEGHNA ACHARJEE

02204092014

Certified that the Project Report entitled VxLAN Troubleshooting Automation done by the
above student is completed under my guidance.

Signature of the Supervisor: Signature of the Faculty Mentor:

Date: Date:
Name of the Supervisor: Name of the Faculty Mentor:
Designation: Designation:
Organisation Name:
 UNDERTAKING REGARDING ANTI-PLAGIARISM

I, hereby, declare that the material/ content presented in the report are free from plagiarism
and is properly cited and written in my own words. In case, plagiarism is detected at any stage,
I shall be solely responsible for it.

MEGHNA ACHARJEE

02204092014
 ACKNOWLEDGEMENTS

As a part of process in the completion of the project titled VxLAN Troubleshooting

Automation, I would like to extend my gratitude to my external guide, Mr. Solomon
Sudhakar, Technical support Engineer, Cisco Systems, Inc. for his unyielding support and
guidance throughout the course of the project. I would also like to extend my gratitude to Ms.
Suruchi Kale and Mr. Kapil Rathi, Cisco Systems, Inc. for their guidance in the mechanics
of the project.

Further, I would like to thank Dr. A.K. Mohapatra (HOD, IT) and my internal guide- Ms.
Ankita Singh (Assistant Professor, IT), for her valuable guidance and timely suggestions
during the entire duration of my seminar work, without which this work would not have been
possible. I would also like to convey my deep regards to all other faculty members of
department IT, who have bestowed their great effort and guidance at appropriate times without
which it would have been very difficult on my part to finish this work.

I would also like to thank my project partner Aditya Gaurav (B.Tech, VIT) for his
contribution to the project. Finally, I would also thank my family and friends for their advice
and moral support.

02204092014

MEGHNA ACHARJEE
 ABSTRACT

Network segmentation services are traditionally provided by Network Architects by setting up

VLANs. VLANs provide logical segmentation of Layer 2 boundaries or broadcast domains.
However, due to the inefficient use of available network links with VLAN use, rigid requirements on
device placements in the data center network, and the limited scalability to a maximum 4094 VLANs,
using VLANs has become a limiting factor to IT departments and cloud providers as they build large
multitenant data centers.

Cisco, in partnership with other leading vendors, proposed the Virtual Extensible LAN (VXLAN)
standard to the IETF, which is documented by the IETF in RFC 7348.

VXLAN solves three main problems:

16M VNIs (broadcast domains) versus the 4K offered by traditional VLANs.

Allows L2 to be extended anywhere in an IP network.
Optimized flooding

If the communication between two hosts is unsuccessful, then the troubleshooting needs to be done.
This troubleshooting has to be automated using a scripting language like Python to,

1) Reduce the time involved in manual troubleshooting there by increasing productivity in Data
Centers.
2) Engineers do not need the expertise or remember complex steps involved in troubleshooting
as the script takes care of the troubleshooting.

The project will involve:

a) Configuring and Troubleshooting Cisco Nexus 9000 Series Switches.
b) Automation of troubleshooting of a VxLAN issue by using Python scripting language.
c) This script will be integrated to CLI analyzer, which is the automation platform for Cisco
TAC.
TABLE OF CONTENTS

(i) CERTIFICATE

(ii) UNDERTAKING REGARDING ANTI-PLAGERISM

(iii) ACKNOWLEDGEMENTS

(iv) ABSTRACT

LIST OF FIGURES .......................................................................................................... 4

LIST OF TABLES ............................................................................................................ 7

LIST OF SYMBOLS ........................................................................................................ 8

CHAPTER 1: INTRODUCTION .................................................................................... 9

1.1 THEORETICAL BACKGROUND ................................................................... 10

1.2 MOTIVATION .................................................................................................. 11

1.3 AIM OF THE PROPOSED WORK .................................................................. 11

1.4 OBJECTIVE OF THE PROPOSED WORK ..................................................... 12

1.5 SCOPE OF PROJECT ....................................................................................... 13

1.6 TIMELINE / GANTT CHART .......................................................................... 13

1.7 ROLE IN PROJECT .......................................................................................... 14

1.8 ORGANIZATION OF THE THESIS ................................................................ 14

CHAPTER 2: SYSTEM ANALYSIS ............................................................................ 15

2.1 SURVEY OF EXISTING MODEL ................................................................... 15

2.2 INTRODUCTION AND RELATED CONCEPTS ........................................... 16

2.3 STAKEHOLDERS OF THE SYSTEM ............................................................. 17

2.4 BUSINESS WORKFLOW ................................................................................ 17

2.5 FUNCTIONAL DESCRIPTION OF THE SYSTEM ........................................ 18

1
 2.5.1 ADMIN ....................................................................................................... 18

2.5.2 CLIENT ...................................................................................................... 18

2.5.3 CISCO TAC ENGINEER ........................................................................... 19

2.6 DATA FLOW DIAGRAM ................................................................................ 20

2.6.1 CONTEXT LEVEL DIAGRAM ................................................................ 20

2.6.2 DFD LEVEL 1 DIAGRAM ........................................................................ 21

2.6.3 DFD LEVEL 2 DIAGRAM ........................................................................ 22

2.7 ENTITY- RELATIONSHIP DIAGRAM .......................................................... 23

CHAPTER 3: SYSTEM DESIGN ................................................................................. 24

3.1 DETERMINATION OF REQUIREMENTS ..................................................... 24

3.2 HARDWARE REQUIREMENTS ..................................................................... 24

3.3 SOFTWARE REQUIREMENTS ...................................................................... 25

3.3.1 PYTHON .................................................................................................... 25

3.3.2 WINSCP ..................................................................................................... 25

3.3.3 PUTTY ........................................................................................................ 26

3.3.4 BDB ............................................................................................................ 26

3.3.5 CISCO CLI ANALYZER ........................................................................... 26

3.4 SYSTEM ARCHITECTURE............................................................................. 27

3.4.1 TOPOLOGY ............................................................................................... 27

3.4.2 PACKET FLOW ......................................................................................... 28

CHAPTER 4: IMPLEMENTATION ........................................................................... 29

4.1 METHODOLOGY ............................................................................................. 29

4.2 MODULES......................................................................................................... 30

4.2.1 CONFIGURATION OF NEXUS 9K SWITCHES .................................... 30

4.2.2 GENERATION OF PYTHON SCRIPTS ................................................... 32

2
 4.2.3 USE OF PUTTY AND WINSCP FOR RUNNING PYTHON SCRIPTS . 33

4.2.4 GENERATION OF BDB SCRIPTS ........................................................... 35

4.2.5 MIGRATION TO CLI ANALYZER APP ................................................. 37

4.3 SIMULATION TOOL FOR USER ................................................................... 38

4.3.1 PYTHON SCRIPTS ON PUTTY ............................................................... 38

4.3.2 BDB SCRIPTS ON CLI ANALYZER ....................................................... 42

4.4 CONCLUSION .................................................................................................. 45

CHAPTER 5: INTERFACE DESIGN .......................................................................... 46

5.1 PYTHON SCRIPT AND PUTTY...................................................................... 46

5.1.1 ENCAP ....................................................................................................... 47

5.1.2 DECAP ....................................................................................................... 50

5.2 BDB SCRIPT AND CLI ANALYZER ............................................................. 52

5.2.1 ENCAP ....................................................................................................... 52

5.2.2 DECAP ....................................................................................................... 62

CHAPTER 6: CONCLUSION AND FUTURE WORK ............................................. 68

6.1 MODULES DEVELOPED ................................................................................ 68

6.2 FUTURE WORK ............................................................................................... 69

REFERENCES ................................................................................................................ 70

3
LIST OF FIGURES

Figure 2.5.1.1 VxLAN Frame Format Courtesy: Cisco.................................................... 10

Figure 2.5.1.1 Block Diagram........................................................................................... 13
Figure 2.5.1.1 Timeline/ Gantt Chart ................................................................................ 13
Figure 2.5.1.1 VTEP Courtesy: Cisco .............................................................................. 16
Figure 2.5.1.1 Flow Chart ................................................................................................. 17
Figure 2.6.1.1 Context Level Diagram ............................................................................. 20
Figure 2.6.2.1 DFD Level 1 Diagram ............................................................................... 21
Figure 2.6.3.1 DFD Level 2 Diagram ............................................................................... 22
Figure 2.6.3.1 Context Level Diagram ............................................................................. 23
Figure 1.4.1.1 Topology.................................................................................................... 27
Figure 1.4.2.1 Packet Flow Courtesy: Cisco..................................................................... 28
Figure 4.2.1.1 Checking Out Nexus 9k Pod ..................................................................... 30
Figure 4.2.1.2 Devices 1 ................................................................................................... 31
Figure 4.2.1.3 Devices 2 ................................................................................................... 31
Figure 4.2.1.4 Devices 3 ................................................................................................... 32
Figure 4.2.3.1 Putty Configuration ................................................................................... 33
Figure 4.2.3.2 Putty Interface ........................................................................................... 33
Figure 4.2.3.3 WinSCP Interface ...................................................................................... 34
Figure 4.2.4.1 BDB Applications Courtesy: Cisco ........................................................... 35
Figure 4.2.4.2 BDB Platform ............................................................................................ 35
Figure 4.2.4.3 BDB Interface ............................................................................................ 36
Figure 4.2.5.1 CLI Analyzer Interface .............................................................................. 37
Figure 4.3.1.1 Telnet to the device ................................................................................... 38
Figure 4.3.1.2 Fill in the login details ............................................................................... 39
Figure 4.3.1.3 Enable sftp ................................................................................................. 39
Figure 4.3.1.4 Open WinSCP and login ........................................................................... 40
Figure 4.3.1.5 WinSCP Layout ......................................................................................... 40
Figure 4.3.1.6 Copy Python Script.................................................................................... 41
Figure 4.3.1.7 Run Script on Putty ................................................................................... 41
4
Figure 4.3.2.1 Open CLI Analyzer ................................................................................... 42
Figure 4.3.2.2 Click on New Session ................................................................................ 43
Figure 4.3.2.3 Click on Form Test Tool ........................................................................... 43
Figure 4.3.2.4 Choose the name of the Script ................................................................... 44
Figure 5.1.1.1 Encap Putty Output 1................................................................................. 47
Figure 5.1.1.2 Encap Putty Output 2................................................................................. 48
Figure 5.1.1.3 Encap Putty Output 3................................................................................. 48
Figure 5.1.1.4 Encap Putty Output 4................................................................................. 49
Figure 5.1.2.1 Decap Putty Output 1 ................................................................................ 50
Figure 5.1.2.2 Decap Putty Output 2 ................................................................................ 51
Figure 5.1.2.3 Decap Putty Output 3 ................................................................................ 51
Figure 5.2.1.1 Encap CLI Analyzer Form ........................................................................ 52
Figure 5.2.1.2 Encap CLI Analyzer Output 1 ................................................................... 53
Figure 5.2.1.3 Encap CLI Analyzer Output 2 ................................................................... 53
Figure 5.2.1.4 Encap CLI Analyzer Output 3 ................................................................... 54
Figure 5.2.1.5 Encap CLI Analyzer Output 4 ................................................................... 54
Figure 5.2.1.6 Encap CLI Analyzer Output 5 ................................................................... 55
Figure 5.2.1.7Encap CLI Analyzer Output 6 .................................................................... 55
Figure 5.2.1.8 Encap CLI Analyzer Output 7 ................................................................... 56
Figure 5.2.1.9 Encap CLI Analyzer Output 8 ................................................................... 56
Figure 5.2.1.10 Encap CLI Analyzer Output 9 ................................................................. 57
Figure 5.2.1.11 Encap CLI Analyzer Output 10 ............................................................... 57
Figure 5.2.1.12 Encap CLI Analyzer Output 11 ............................................................... 58
Figure 5.2.1.13 Encap CLI Analyzer Output 12 ............................................................... 58
Figure 5.2.1.14 Encap CLI Analyzer Output 13 ............................................................... 59
Figure 5.2.1.15 Encap CLI Analyzer Output 14 ............................................................... 59
Figure 5.2.1.16 Encap CLI Analyzer Output 15 ............................................................... 60
Figure 5.2.1.17 Encap CLI Analyzer Output 16 ............................................................... 60
Figure 5.2.1.18 Encap CLI Analyzer Output 17 ............................................................... 61
Figure 5.2.1.19 Encap CLI Analyzer Output 18 ............................................................... 61
Figure 5.2.2.1 Decap CLI Analyzer Form ........................................................................ 62

5
Figure 5.2.2.2 Decap CLI Analyzer Output 1 ................................................................... 63
Figure 5.2.2.3 Decap CLI Analyzer Output 2 ................................................................... 63
Figure 5.2.2.4 Decap CLI Analyzer Output 3 ................................................................... 64
Figure 5.2.2.5 Decap CLI Analyzer Output 4 ................................................................... 64
Figure 5.2.2.6 Decap CLI Analyzer Output 5 ................................................................... 65
Figure 5.2.2.7 Decap CLI Analyzer Output 6 ................................................................... 65
Figure 5.2.2.8 Decap CLI Analyzer Output 7 ................................................................... 66
Figure 5.2.2.9 Decap CLI Analyzer Output 8 ................................................................... 66
Figure 5.2.2.10 Decap CLI Analyzer Output 9 ................................................................. 67
Figure 5.2.2.11 Decap CLI Analyzer Output 10 ............................................................... 67

6
LIST OF TABLES

Table 2.5.1-1 Functional Description of Admin .............................................................. 18

Table 2.5.2-1 Functional Description of Client ............................................................... 18
Table 2.5.3-1 Functional Description of Cisco TAC Engineer ....................................... 19
Table 2.6.3-1 Hardware Requirements ............................................................................. 24

7
LIST OF SYMBOLS

1. NV- Network Virtualization

2. VxLAN- Virtual Extensible LAN

3. VTEP- Virtual Tunnel End Point

4. vPC- Virtual Port Channel

5. PIM- Protocol Independent Multicast

6. CLI- Command Line Interface

8
 VXLAN TROUBLESHOOTING AUTOMATION

CHAPTER 1: INTRODUCTION

Cisco Systems, Inc. is an American corporation technology company headquartered in

San Jose, California, that designs, manufactures and sells networking equipment
worldwide. It is the largest networking company in the world.

The Cisco Technical Assistance Center (TAC) provides around-the-clock, award-

winning technical support services for Cisco products and technologies.
www.cisco.com/tac. Cisco Live is a powerful suite of browser-based support tools that
allows Cisco customers to collaborate over the Internet with Cisco TAC engineers.

VxLAN is an upcoming technology and there are many clients who open TAC cases for
any sort of issue. The TAC Engineer needs to memorise commands (or refer to a
document) and run those commands on the users switches. The commands are not
simple or a copy paste job. The output of the previous command is then used as input of
the next command. Hence, the TAC Engineer needs to be very careful because even a
single mistake can create many issues.

To decrease human effort and the chances of human error, I have tried to automate the
troubleshooting script. Now, the TAC Engineer only needs some basic information from
the client, fill up the form and come to know about the possible issue within minutes.

9
 VXLAN TROUBLESHOOTING AUTOMATION

1.1 THEORETICAL BACKGROUND

VXLAN is a MAC in IP/UDP (MAC-in-UDP) encapsulation technique with a 24-bit

segment identifier in the form of a VXLAN ID. The larger VXLAN ID allows LAN
segments to scale to 16 million in a cloud network. In addition, the IP/UDP encapsulation
allows each LAN segment to be extended across existing Layer 3 networks making use of
Layer 3 equal-cost multipath (ECMP). Cisco Nexus 7000 switches are designed for
hardware-based VXLAN function. This can enable virtualized and multi-tenant data
centre designs over a shared common physical infrastructure. VXLAN provides a way to
extend Layer 2 networks across Layer 3 infrastructure using MAC-in-UDP encapsulation
and tunnelling. VXLAN enables flexible workload placements using the Layer 2
extension. It can also be an approach to building a multi-tenant data centre by decoupling
tenant Layer 2 segments from the shared transport network.When deployed as a VXLAN
gateway, Cisco Nexus 7000 switches can connect VXLAN and classic VLAN segments
to create a common forwarding domain so that tenant devices can reside in both
environments. VXLAN has the following benefits:

Flexible placement of multi-tenant segments throughout the data centre. It provides a

way to extend Layer 2 segments over the underlying shared network infrastructure so
that tenant workloads can be placed across physical pods in the data centre.
Higher scalability to address more Layer 2 segments.

Figure 2.5.1.1 VxLAN Frame Format Courtesy: Cisco

10
 VXLAN TROUBLESHOOTING AUTOMATION

1.2 MOTIVATION

VxLAN is designed to provide the same Ethernet Layer 2 network services as VLAN
does today, but with greater extensibility and flexibility. It provides 16M VNIs (broadcast
domains) versus the 4K offered by traditional VLANs.

Further, the troubleshooting needs to be done in the case of any issue. But for
troubleshooting, the Network Engineer needs to remember complex commands and run
them manually. This is very time-consuming. If this process is automated, the problem
can be solved faster and more efficiently.

1.3 AIM OF THE PROPOSED WORK

The VxLan Topology will be set up using the Cisco Switches, which will be configured
from scratch. The troubleshooting of possible issues will be done and a script will be
written in Python to automate the troubleshooting. Finally, the script will be modified
into a BDB script and migrated to CLI Analyzer.

11
 VXLAN TROUBLESHOOTING AUTOMATION

1.4 OBJECTIVE OF THE PROPOSED WORK

Network segmentation services are traditionally provided by Network Architects by

setting up VLANs. VLANs provide logical segmentation of Layer 2 boundaries or
broadcast domains. However, due to the inefficient use of available network links with
VLAN use, rigid requirements on device placements in the data centre network, and the
limited scalability to a maximum 4094 VLANs, using VLANs has become a limiting
factor to IT departments and cloud providers as they build large multitenant data centres.

Cisco, in partnership with other leading vendors, proposed the Virtual Extensible LAN
(VXLAN) standard to the IETF, which is documented by the IETF in RFC 7348.

VXLAN solves three main problems:

16M VNIs (broadcast domains) versus the 4K offered by traditional VLANs.

Allows L2 to be extended anywhere in an IP network.
Optimized flooding

If the communication between two hosts is unsuccessful, then the troubleshooting needs
to be done.

The main objective of the project is to understand VxLAN and to set it up and automate
its troubleshooting. The script would be used by Cisco TAC Engineers in their problem-
solving.

12
 VXLAN TROUBLESHOOTING AUTOMATION

1.5 SCOPE OF PROJECT

Here is a block diagram showing the scope of the project.

Admin A Configuration

Troubleshooting TAC Engineer

Client

Figure 2.5.1.1 Block Diagram

1.6 TIMELINE / GANTT CHART

The project started on 9th January 2017 and is expected to finish in 16 weeks.

Figure 2.5.1.1 Timeline/ Gantt Chart

13
 VXLAN TROUBLESHOOTING AUTOMATION

1.7 ROLE IN PROJECT

We have 2 members in this project team, me and Aditya Gaurav from VIT. I am the
Team Leader. My duties include creating the work schedule for the day, work
distribution amongst the team, documentation and interaction with the company mentor
on a daily basis. The actual project work is equally divided amongst the members.

1.8 ORGANIZATION OF THE THESIS

Chapter 1 deals with the introduction.

Chapter 2 will include the proposed system, DFD and ER diagrams.
Chapter 3 will include the system design.
Chapter 4 covers the implementation methodology and modules.
Chapter 5 will contain the screenshots of the interfaces.
Chapter 6 will contain the conclusion and future work.

14
 VXLAN TROUBLESHOOTING AUTOMATION

CHAPTER 2: SYSTEM ANALYSIS

In this chapter, I will discuss the proposed system.

The proposed system would be a BDB Script migrated on CLI Analyzer to automate
troubleshooting of VxLANs.

2.1 SURVEY OF EXISTING MODEL

A VLAN is a group of devices on one or more LANs that are configured to communicate
as if they were attached to the same wire when in fact, they are located on a number of
different LAN segments. Because VLANs are based on logical instead of physical
connections, they are extremely flexible.

But, VLANs provide limited scalability. Thus, we are moving towards VxLANs for an
improved scalability.

The troubleshooting procedure for VxLANs is currently done manually.

For Layer 2:

1. Check if both the source and destination VLAN IDs are same.
2. Check if the vrf is mentioned.
3. Check if nve peers are obtained.
4. Check if vrf is same.
5. Run various L2 commands for troubleshooting. (Both Encap and Decap)

For Layer 3:

1. Check if both the source and destination VLAN IDs are different.
2. Check if the vrf is mentioned.
3. Check if nve peers are obtained.
4. Check if vrf is same.

15
 VXLAN TROUBLESHOOTING AUTOMATION

5. Run various L2 commands for troubleshooting. (Both Encap and Decap)

6. Run various L3 commands for troubleshooting. (Both Encap and Decap)

2.2 INTRODUCTION AND RELATED CONCEPTS

With growing adoption of virtualization in the customer environment and a large number
of workload mobility requirements in the data centre; overlays are becoming a key
technology. VXLAN is one the overlay technology.

1. Overlay: An overlay network is a computer network that is built on top of

another network. Nodes in the overlay network can be thought of as being
connected by virtual or logical links, each of which corresponds to a path, perhaps
through many physical links, in the underlying network.

2. VxLAN: Virtual Extensible LAN (VXLAN) is a network virtualization

technology that attempts to improve the scalability problems associated with large
cloud computing deployments.

3. VTEP: Frame encapsulation is done by an entity known as a VXLAN Tunnel

Endpoint (VTEP.) A VTEP has two logical interfaces: an uplink and a downlink.
The uplink is responsible for receiving VXLAN frames and acts as a tunnel
endpoint with an IP address used for routing VXLAN encapsulated frames.

Figure 2.5.1.1 VTEP Courtesy: Cisco

16
 VXLAN TROUBLESHOOTING AUTOMATION

2.3 STAKEHOLDERS OF THE SYSTEM

Admin: The admin is responsible for setting up the VxLAN topology and making
the customer aware of all the services provided by Cisco.

Client: The client is the end user/ network engineer of the end user. They will
discuss their issue with the system and cooperate in the troubleshooting by
providing necessary information.

Cisco TAC Engineer: The TAC engineer does the actual troubleshooting. They
will run the BDB Script on CLI Analyzer (on the clients devices) on the basis of
the information provided. On the basis of the outputs, they will determine the
issue and provide solutions.

2.4 BUSINESS WORKFLOW

The flow of all the processes involved in this project is given below:

Standalone
Configure
Learn VxLAN Troubleshoot troubleshooting
VxLAN on N9K
python script

Migrate
Practise BDB and
standalone N9K Test
CLI
script

Figure 2.5.1.1 Flow Chart

17
 VXLAN TROUBLESHOOTING AUTOMATION

2.5 FUNCTIONAL DESCRIPTION OF THE SYSTEM

The functions and responsibilities of each of the stakeholders is mentioned in this section.

2.5.1 ADMIN

S.NO. RESPONSIBILITIES
1. Gather Client Requirements
2. Set up topology.
3. Optimization of system.
4. Handing over system to client.
5. Making client aware of Cisco Services.

Table 2.5.1-1 Functional Description of Admin

2.5.2 CLIENT

S.NO. RESPONSIBILITIES
1. Provide requirements.
2. Cooperate in setting up topology.
3. Being aware of Cisco Services.
4. Inform TAC of any issues.
5. Open a TAC Case.
6. Cooperate with TAC in troubleshooting.

Table 2.5.2-1 Functional Description of Client

18
 VXLAN TROUBLESHOOTING AUTOMATION

2.5.3 CISCO TAC ENGINEER

S.NO. RESPONSIBILITIES
1. Listen to Client issues.
2. Get knowledge of Client topology.
3. Run troubleshooting script on Client devices
4. Identify issue.
5. Solve issue.

Table 2.5.3-1 Functional Description of Cisco TAC Engineer

19
 VXLAN TROUBLESHOOTING AUTOMATION

2.6 DATA FLOW DIAGRAM

The data flow diagram represents the flow of information among the various processes of
the system. The context level diagram defines the scope of the system in term of various
responsibilities bestowed to the system. Level 1 Diagram shows the major subsystems
and the interaction among them.

2.6.1 CONTEXT LEVEL DIAGRAM

The overall scope of the system is represented through context level diagram. The system
interacts with stakeholder for accomplishing the various requirements for execution of
the system.

Show Commands
Client 0

Nexus Switch Configuration

and Troubleshooting
Admin
Regex

TAC Engineer

Figure 2.6.1.1 Context Level Diagram

20
 VXLAN TROUBLESHOOTING AUTOMATION

2.6.2 DFD LEVEL 1 DIAGRAM

The major components of the system are Topology Configuration, Troubleshooting,

Creating a Python Script, BDB Script and migrating the script to CLI Analyzer.

1.0
Admin
Topology Configuration

Regex
Client 2.0

Troubleshooting

Show Commands

3.0

Python Script

4.0

BDB Script

5.0
TAC Engineer
CLI Analyzer

Figure 2.6.2.1 DFD Level 1 Diagram

21
 VXLAN TROUBLESHOOTING AUTOMATION

2.6.3 DFD LEVEL 2 DIAGRAM

The major system is troubleshooting. It is represented through a Level 2 DFD Diagram.

2.1
Client Encapsulation

Regex
2.2

Decapsulation

Show Commands

2.3 2.4
L2 Verification
L3 Verification

2.5
Generation of
Script

Figure 2.6.3.1 DFD Level 2 Diagram

22
 VXLAN TROUBLESHOOTING AUTOMATION

2.7 ENTITY- RELATIONSHIP DIAGRAM

An entity relationship diagram (ERD) shows the relationships of entity sets stored in a
database. An entity in this context is a component of data.

Nexus 9k Switches

Topology
Encapsulation Decapsulation

CONFIGURATION
Show
TROUBLESHOOTING
Commands

PYTHON AND
Executes
BDB SCRIPTS

Runs
CLI ANALYSER

Figure 2.6.3.1 Context Level Diagram

23
 VXLAN TROUBLESHOOTING AUTOMATION

CHAPTER 3: SYSTEM DESIGN

Troubleshooting is an important step in development. If this troubleshooting can be

automated, it will save time and money of both the company and the client. This chapter
will cover the topology considered and the tools used.

3.1 DETERMINATION OF REQUIREMENTS

The first requirement is the availability of pod of Nexus 9k switches. Then we need to
configure the switches and set up the topology according to our requirements. VxLAN
needs to be set up and checked using show nve peers command. We will also be
requiring tools to access the devices (for configuration and troubleshooting). Finally, we
need tools and platforms to develop the troubleshooting automation script and to migrate
it to a standalone, user-friendly app.

3.2 HARDWARE REQUIREMENTS

Platform NX-OS version

Minimum Recommended
Nexus 9500 6.1.2I3.1.x 6.1.2.I3.1.x
Nexus 9300 6.1.2I2.1.x 6.1.2.I2.1.x
Nexus 3100 (3132/3172) 6.x Q2 CY14
Nexus 6000 7.0(0)N1x) Q3 CY14
Nexus 7000 with F3 7.0.x Q4 CY14

Table 2.6.3-1 Hardware Requirements

24
 VXLAN TROUBLESHOOTING AUTOMATION

3.3 SOFTWARE REQUIREMENTS

The following softwares are used in this project:

Python
WinSCP
Putty
BDB
Cisco CLI Analyzer

3.3.1 PYTHON

Python is a scripting language. It is a high level programming language. It is widely

used in developing web apps, games etc. It is an interpreted language.

In this project, python is used to develop troubleshooting scripts and then regular
expressions are used to extract interesting information. These scripts are run on the
devices.

3.3.2 WINSCP

WinSCP (Windows Secure Copy) is a free and open source SFTP, FTP, WebDAV and
SCP client for Microsoft Windows. Its main function is secure file transfer between a
local and a remote computer. Beyond this, WinSCP offers basic file manager and file
synchronization functionality. For secure transfers, it uses Secure Shell (SSH) and
supports the SCP protocol in addition to SFTP.

In this project, we used the SFTP server to transfer the Python scripts to the devices.

25
 VXLAN TROUBLESHOOTING AUTOMATION

3.3.3 PUTTY

PuTTY is a client program for the SSH, Telnet and Rlogin network protocols. These
protocols are all used to run a remote session on a computer, over a network. It is a free
and open-source terminal emulator, serial console and network file transfer application.

In this project, we use Putty to access the device remotely. We can configure the device
and run the python scripts on the device. WinSCP is compatible with Putty.

3.3.4 BDB

BDB is a scripting platform provided by Cisco. It can only be accessed by someone with
a Cisco ID. Scripts can be created in various languages such as Python, Perl etc.
However, syntax of BDB needs to be followed. BDB is similar to an appstore. People can
upload new scripts and use scripts written by themselves or others.

In this project, we use the BDB platform to create a standalone script. Python is the
scripting language which is used.

3.3.5 CISCO CLI ANALYZER

The Cisco CLI Analyzer (formerly ASA CLI Analyzer) is a smart SSH client with
internal TAC tools and knowledge integrated. It is designed to help troubleshoot and
check the overall health of your Cisco supported software

In this project, the internal Cisco TAC version was used to migrate the BDB Script to
CLI Analyzer.

26
 VXLAN TROUBLESHOOTING AUTOMATION

3.4 SYSTEM ARCHITECTURE

System Architecture is the fundamental organization of a system embodied in its

components, their relationships to each other and to the environment and the principles
guiding its design and evolution.

3.4.1 TOPOLOGY

Figure 3.4.1.1 Topology

9508-A acts as the spine.

9396-A is VTEP 1.
9396-B is VTEP 2.

27
 VXLAN TROUBLESHOOTING AUTOMATION

9396-C is VTEP 3.
9396-D is VTEP 4.
5672-A and 5672-B are for the traffic flow.

There is vPC (Virtual Port Channel configured between VTEP 1 and VTEP 2). It acts as
an EtherChannel and HSRP both.

Encapsulation takes place on the VTEP 3 and VTEP 4 side.

Decapsulation takes place on the VTEP 1 and VTEP 2 side.

If Vlan ID is the same we do Layer 2 verification.

If VLAN IDs are different, we do Layer 3 verification.

3.4.2 PACKET FLOW

Figure 3.4.2.1 Packet Flow Courtesy: Cisco

28
 VXLAN TROUBLESHOOTING AUTOMATION

CHAPTER 4: IMPLEMENTATION

In this chapter I will discuss the methodology, modules and give a user guide for user
interaction.

4.1 METHODOLOGY

1. Complete Requirements Gathering

a. To get a clear of the project and identify the final product. Also become
aware of the rules and regulations of the workplace.

2. Study VxLAN
a. Go through videos and documents available to get to know the VxLAN
technology. Getting familiar with important terminologies of VxLAN. Get
to know about various methods on which VxLAN works.

3. Configure VxLAN on Cisco N9K switch

a. Configure the required topology using Cisco Nexus 9K switches. Try to
mess up and then fix it back in order to get an in-depth knowledge.

4. Troubleshoot VxLAN on N9K switch

a. Try to bring the topology into working state again, which is down.

5. Standalone Troubleshooting script for N9K

a. Write a script which is capable of troubleshooting the VxLAN topology
made of N9K switches using python.

6. Get Hands on BDB server and CLI analyzer

a. BDB and CLI are Cisco Systems, Inc. proprietary platforms which are
used to support various technologies. Try to get hands on, in order to use it
to upload the scripts and make the whole process public.

29
 VXLAN TROUBLESHOOTING AUTOMATION

7. Migrate standalone scripts to CLI analyzer

a. Migrate the standalone scripts written to BDB servers and do testing.

4.2 MODULES

The project is divided into 5 major modules:

Configuration of Nexus 9k Switches

Generation of Python Scripts
Use of Putty and WinSCP for running the Python Scripts
Generation of BDB Scripts
Migration to CLI Analyzer App

4.2.1 CONFIGURATION OF NEXUS 9K SWITCHES

The switches are available virtually on Cisco CALO site. This site is only accessible by
people with a Cisco ID. The required pod needs to be issued and then used. Nexus 9k
switches are used and they are configured from scratch. We need to use a software to
Telnet or SSH to the switch (Putty and CLI Analyzer).

Figure 4.2.1.1 Checking Out Nexus 9k Pod

30
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 4.2.1.2 Devices 1

Figure 4.2.1.3 Devices 2

31
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 4.2.1.4 Devices 3

In total, 9 devices need to be configured for the VxLAN Topology we have considered.
These scripts will be stored in the bootflash of each device and reloaded when required.
The devices are accessed remotely by Telnet/SSH using Putty.

4.2.2 GENERATION OF PYTHON SCRIPTS

The Hardware and software verification commands were studied for L2 (same VLAN)
and L3 (different VLANs). The output of one command is used as input to another
command. The significant part of the output were extracted using regular expressions
and string matching. Then, two python scripts were created for:

Encapsulation (Both L2 and L3)

Decapsulation (Both L2 and L3)

The scripts automate the troubleshooting process.

32
 VXLAN TROUBLESHOOTING AUTOMATION

4.2.3 USE OF PUTTY AND WINSCP FOR RUNNING PYTHON SCRIPTS

Putty is a tool used to Telnet/ SSH to remotely access a device.

Figure 4.2.3.1 Putty Configuration

We can access the device and enable stft-server in the devices.

Figure 4.2.3.2 Putty Interface

33
 VXLAN TROUBLESHOOTING AUTOMATION

WinSCP is a sftp-server. It is used to load the python script into the device

Figure 4.2.3.3 WinSCP Interface

Finally, the script is run on Putty.

Encapsulation script is run on VTEP 3 and VTEP 4.

Decapsulation script is run on VTEP 1 and VTEP 2.

The point where the script fails, is the point of error in the topology and that part would
need to be troubleshooted or corrected.

34
 VXLAN TROUBLESHOOTING AUTOMATION

4.2.4 GENERATION OF BDB SCRIPTS

BDB is a scripting platform provided by Cisco. It can only be accessed by someone with
a Cisco ID. Scripts can be created in various languages such as Python, Perl etc.
However, syntax of BDB needs to be followed. BDB is similar to an appstore. People can
upload new scripts and use scripts written by themselves or others. Except it is free. It is a
community where people started to collaborate in writing scripts across borders. Many
scripts today are written in teams of coders, several of them having coders in different
parts of the world.

Open APIs

A scripts app External

store libraries

Automation
on SR
attachments

A community Custom
! integrations

Figure 4.2.4.1 BDB Applications Courtesy: Cisco

Figure 4.2.4.2 BDB Platform

35
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 4.2.4.3 BDB Interface

Then, two standalone BDB scripts were created for:

Encapsulation (Both L2 and L3)

Decapsulation (Both L2 and L3)

The scripts automate the troubleshooting process.

36
 VXLAN TROUBLESHOOTING AUTOMATION

4.2.5 MIGRATION TO CLI ANALYZER APP

The Cisco CLI Analyzer (formerly ASA CLI Analyzer) is a smart SSH client with internal
TAC tools and knowledge integrated. It is designed to help troubleshoot and check the
overall health of your Cisco supported software

The internal Cisco TAC version was used to migrate the BDB Script to CLI Analyzer.

Figure 4.2.5.1 CLI Analyzer Interface

Encapsulation script is run on VTEP 3 and VTEP 4.

Decapsulation script is run on VTEP 1 and VTEP 2.

The point where the script fails, is the point of error in the topology and that part would
need to be troubleshooted or corrected.

37
 VXLAN TROUBLESHOOTING AUTOMATION

4.3 SIMULATION TOOL FOR USER

The uses may run the troubleshooting in two ways:

Python Scripts on Putty

BDB Scripts on CLI Analyzer

Encapsulation script is run on VTEP 3 and VTEP 4.

Decapsulation script is run on VTEP 1 and VTEP 2.

The point where the script fails, is the point of error in the topology and that part would
need to be troubleshooted or corrected.

4.3.1 PYTHON SCRIPTS ON PUTTY

Here is a user guide for the user to run the Python Scripts on Putty:

1. Telnet to the device using Putty.

Figure 4.3.1.1 Telnet to the device

38
 VXLAN TROUBLESHOOTING AUTOMATION

2. Fill in the login details.

Figure 4.3.1.2 Fill in the login details

3. Enable feature sftp-server.

Figure 4.3.1.3 Enable sftp

39
 VXLAN TROUBLESHOOTING AUTOMATION

4. Open WinSCP and connect to the device.

Figure 4.3.1.4 Open WinSCP and login

5. On the left are the files on your computer and on the right are the files on the
device.

Figure 4.3.1.5 WinSCP Layout

40
 VXLAN TROUBLESHOOTING AUTOMATION

6. Go to Scripts and copy your python script.

Figure 4.3.1.6 Copy Python Script

7. Run the script on device (source [filename.py]) using Putty.

Figure 4.3.1.7 Run Script on Putty

41
 VXLAN TROUBLESHOOTING AUTOMATION

4.3.2 BDB SCRIPTS ON CLI ANALYZER

Here is a user guide for the user to use the CLI Analyzer App:

Open CLI Analyzer.

Figure 4.3.2.1 Open CLI Analyzer

42
 VXLAN TROUBLESHOOTING AUTOMATION

Go to 'New Session' on the left hand menu and log into your device.

Figure 4.3.2.2 Click on New Session

From the tools menu choose 'Form Test Tool'.

Figure 4.3.2.3 Click on Form Test Tool

43
 VXLAN TROUBLESHOOTING AUTOMATION

Choose the name of the script and click on continue.

Figure 4.3.2.4 Choose the name of the Script

These are the parameters that the user needs to input :

Source VLAN ID: The VLAN ID of the source from where the packet is coming.
Destination VLAN ID: The VLAN ID of the destination.
Source IP Address: The IP Address of the source host.
Destination IP Address: The IP Address of the destination host.
Source MAC Address: The MAC Address of the source host.
Destination MAC Address: The MAC Address of the destination host.

After these inputs are entered, the Layer 2 hardware verification will take place and if the
VLANs are different, i.e. the source and destination are in different networks then the
Layer 3 hardware verification will take place.

44
 VXLAN TROUBLESHOOTING AUTOMATION

4.4 CONCLUSION

In view of the requirements, all the modules are properly developed, implemented,
optimized and tested. A sample topology is set up and the scripts are made according to
the documentation. The scripts are then tested on different topologies. The bugs are
removed and the scripts are optimized.

Finally, a Cisco Jive documentation is prepared. This serves as a User Manual.

This project has been submitted in TACode 3.0 which is an international competition in
Cisco. The success of this project would make the lives of both clients and Engineers
much easier and would make troubleshooting less complex and less time consuming.

45
 VXLAN TROUBLESHOOTING AUTOMATION

CHAPTER 5: INTERFACE DESIGN

This chapter will show the screenshots of the output obtained.

Two types of scripts are created:

Python Script
BDB Script

Python Script is run on Putty.

BDB Script is run on CLI Analyzer.

5.1 PYTHON SCRIPT AND PUTTY

Two Python Scripts are created:

Encap
Decap

The python scripts are run on Putty.

46
 VXLAN TROUBLESHOOTING AUTOMATION

5.1.1 ENCAP

Encap script is run on VTEP 3 and VTEP 4.

L2 verification is done when VLAN IDs are same.

L2 and L3 verification both are done when VLAN IDs are different.

Figure 5.1.1.1 Encap Putty Output 1

47
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.1.1.2 Encap Putty Output 2

Figure 5.1.1.3 Encap Putty Output 3

48
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.1.1.4 Encap Putty Output 4

49
 VXLAN TROUBLESHOOTING AUTOMATION

5.1.2 DECAP

Decap script is run on VTEP 1 and VTEP 2.

L2 verification is done when VLAN IDs are same.

L2 and L3 verification both are done when VLAN IDs are different.

Figure 5.1.2.1 Decap Putty Output 1

50
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.1.2.2 Decap Putty Output 2

Figure 5.1.2.3 Decap Putty Output 3

51
 VXLAN TROUBLESHOOTING AUTOMATION

5.2 BDB SCRIPT AND CLI ANALYZER

Two BDB Scripts are created:

Encap
Decap

The python scripts are run on CLI Analyzer.

5.2.1 ENCAP

Encap script is run on VTEP 3 and VTEP 4.

L2 verification is done when VLAN IDs are same.

L2 and L3 verification both are done when VLAN IDs are different.

Figure 5.2.1.1 Encap CLI Analyzer Form

52
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.2 Encap CLI Analyzer Output 1

Figure 5.2.1.3 Encap CLI Analyzer Output 2

53
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.4 Encap CLI Analyzer Output 3

Figure 5.2.1.5 Encap CLI Analyzer Output 4

54
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.6 Encap CLI Analyzer Output 5

Figure 5.2.1.7Encap CLI Analyzer Output 6

55
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.8 Encap CLI Analyzer Output 7

Figure 5.2.1.9 Encap CLI Analyzer Output 8

56
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.10 Encap CLI Analyzer Output 9

Figure 5.2.1.11 Encap CLI Analyzer Output 10

57
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.12 Encap CLI Analyzer Output 11

Figure 5.2.1.13 Encap CLI Analyzer Output 12

58
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.14 Encap CLI Analyzer Output 13

Figure 5.2.1.15 Encap CLI Analyzer Output 14

59
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.16 Encap CLI Analyzer Output 15

Figure 5.2.1.17 Encap CLI Analyzer Output 16

60
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.1.18 Encap CLI Analyzer Output 17

Figure 5.2.1.19 Encap CLI Analyzer Output 18

61
 VXLAN TROUBLESHOOTING AUTOMATION

5.2.2 DECAP

Decap script is run on VTEP 1 and VTEP 2.

L2 verification is done when VLAN IDs are same.

L2 and L3 verification both are done when VLAN IDs are different.

Figure 5.2.2.1 Decap CLI Analyzer Form

62
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.2.2 Decap CLI Analyzer Output 1

Figure 5.2.2.3 Decap CLI Analyzer Output 2

63
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.2.4 Decap CLI Analyzer Output 3

Figure 5.2.2.5 Decap CLI Analyzer Output 4

64
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.2.6 Decap CLI Analyzer Output 5

Figure 5.2.2.7 Decap CLI Analyzer Output 6

65
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.2.8 Decap CLI Analyzer Output 7

Figure 5.2.2.9 Decap CLI Analyzer Output 8

66
 VXLAN TROUBLESHOOTING AUTOMATION

Figure 5.2.2.10 Decap CLI Analyzer Output 9

Figure 5.2.2.11 Decap CLI Analyzer Output 10

67
 VXLAN TROUBLESHOOTING AUTOMATION

CHAPTER 6: CONCLUSION AND FUTURE WORK

The Cisco Technical Assistance Center (Cisco TAC) provides around-the-clock, award-
winning technical support services for Cisco products and technologies. TAC has to deal
with a large number of troubleshooting cases daily. This troubleshooting if automated
using a scripting language like Python will,

1) Reduce the time involved in manual troubleshooting there by increasing

productivity in Data Centers.
2) Help Engineers so as they do not need the expertise or remember complex steps
involved in troubleshooting as the script takes care of the troubleshooting.

6.1 MODULES DEVELOPED

The project is divided into 5 major modules:

Configuration of Nexus 9k Switches

Generation of Python Scripts (Encap and Decap for L2 and L3 verification)
Use of Putty and WinSCP for running the Python Scripts
Generation of BDB Scripts (Encap and Decap for L2 and L3 verification)
Migration to CLI Analyzer App

The point of break of the scripts or the point of error is the place which needs to be
concentrated upon.

68
 VXLAN TROUBLESHOOTING AUTOMATION

6.2 FUTURE WORK

This project has been selected for TACode 3.0 which is an international competition
within Cisco. This project will have an international implication and will be used to solve
VxLAN issues worldwide.

The script currently points out to only the place of error. In future, the script should be
able to point out the exact error and correct it. Currently, that part is still done manually.
Full automation will be achieved when human effort is completely removed.

69
 VXLAN TROUBLESHOOTING AUTOMATION

REFERENCES

1. https://techzone.cisco.com/t5/Nexus-9300/MP-BGP-EVPN-VXLAN-L2-Unicast-
Hardware-Verification/ta-p/833812 accessed on Monday 27th March 2017, 5:30 pm

2. https://techzone.cisco.com/t5/Nexus-9300/MP-BGP-EVPN-VXLAN-L3-Unicast-
Hardware-Troubleshooting/ta-p/845647 accessed on Wednesday 29th March 2017, 8:30
pm

3. http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-
switches/white-paper-c11-729383.html accessed on Friday14th April 2017, 2:30 pm

4. http://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-
switches/118978-config-vxlan-00.html accessed on Thursday 20th March 2017, 7:00 pm

5. http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps13386/white- paper-
c11-729383_ns1261_Networking_Solutions_White_Paper.html accessed on
Wednesday15th March 2017, 5:30 pm

70