Versa: uCPE[1] - How to Provision it in 20.

Purushothaman Balakrishnan| Parbhat Kapoor

puru@versa-networks.com
parbhat@versa-networks.com

1 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Purpose/Audience:

1. This document is for Comcast usage only and not meant to be shared externally.

2. Bugs/PR which are mentioned in this document are just for reference/knowledge only and most likely those bugs will be
fixed in 20.2 FRS release.

3. Service-Chain association method is different in R2 and 20.X release. Please refer release notes for further info.

2 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Provisioning v220 as uCPE Device: Creating a bootable USB Drive

1. Create a bootable USB drive

As I was using Windows machine, the only recommended Versa method is by using rufus app.
Please download “rufus” @ here: https://rufus.ie/
Please refer Versa support page for additional info if you are using non-windows machine: https://support.versa-networks.com/support/solutions/articles/23000008849-advantech-install-flexvnf-using-usb

2. For v220 hardware we need to use “wsm.iso” based image:

3. Open “rufus” application and insert a USB drive with your laptop.

4. “rufus” app will auto-detect the USB drive.

5. Select “Disk or ISO” from Boot selection drop down menu.

6. Leave Partition as “MBR” & Target System as “BIOS” as it is.

7. Select “Start”

3 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Provisioning v220 as uCPE Device: Creating a bootable USB Drive

1. Power on the hardware.

2. Please insert USB drive in Slot 1 and follow step 2 as outlined in below url:
https://support.versa-networks.com/support/solutions/articles/23000008849-advantech-install-flexvnf-using-usb

*If you receive “Mounting” related error, please refer below url to mitigate the issue:
https://support.versa-networks.com/support/solutions/articles/23000009663-bootable-usb-retry-mounting-the-cd-r
om-error

4 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Verification:
admin@versa-flexvnf-cli> show interfaces brief | tab
NAME MAC OPER ADMIN TENANT VRF IP
------------------------------------------------------------------------------------------------
eth-0/0 08:35:71:ec:e1:83 down up 0 global 10.10.10.10/8
tvi-0/0 n/a up up - -
tvi-0/0.0 n/a up up 1 Versa-Provider-Control-VR
tvi-0/1 n/a up up - -
tvi-0/1.0 n/a up up 1 Versa-Provider-Control-VR 10.0.33.248/24
vni-0/0 08:35:71:ec:e1:7c up up - - Default config with vni0/0(Physical Port “0”) in up
vni-0/0.0 08:35:71:ec:e1:7c up up 1 WAN1-Transport-VR 192.168.2.6/24 state and receiving DHCP ip address from my Home
vni-0/0.1 08:35:71:ec:e1:7c up up 1 WAN1-Transport-VR Internet Gateway
vni-0/1 08:35:71:ec:e1:7d down up - -
vni-0/1.0 08:35:71:ec:e1:7d down up 1 global 192.168.1.1/24
vni-0/100 02:aa:bb:cc:dd:01 down up - -
vni-0/100.0 02:aa:bb:cc:dd:01 down up 1 WAN1-Transport-VR
vni-0/101 02:aa:bb:cc:dd:02 down up - -
vni-0/101.0 02:aa:bb:cc:dd:02 down up 1 WAN1-Transport-VR
vni-0/102 02:aa:bb:cc:dd:03 down down - -
vni-0/103 02:aa:bb:cc:dd:04 down down - -
vni-0/2 08:35:71:ec:e1:7e down up - -
vni-0/2.0 08:35:71:ec:e1:7e down up 1 WAN1-Transport-VR
vni-0/2.1 08:35:71:ec:e1:7e down up 1 WAN1-Transport-VR
vni-0/3 08:35:71:ec:e1:7f down down - -
vni-0/4 08:35:71:ec:e1:80 down down - -
vni-0/5 08:35:71:ec:e1:81 down up - -
vni-0/5.0 08:35:71:ec:e1:81 down up 1 WAN1-Transport-VR
vni-0/5.1 08:35:71:ec:e1:81 down up 1 WAN1-Transport-VR
vni-0/6 08:35:71:ec:e1:82 down down - -

[ok][2019-01-25 18:43:44]
admin@versa-flexvnf-cli>

5 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
1. Install Hypervisor Packages:

a. For
For greenfield
greenfield based
based uCPE
uCPE deployment
deployment please
please execute
execute below
below command:
a. command:
“request
“request system
system hypervisor
hypervisor enable
enable no-confirm”
no-confirm”
-The
-The above
above command
command will
will create
create the
the OVS-bridges
OVS-bridges &
& interfaces
interfaces from
from vni-0/300
vni-0/300 to
to vni-0/309.
vni-0/309. And
And if
if this
this command
command is
is not
not
executed then installation of 3 rd Party VNF will not go through.
rd
executed then installation of 3 Party VNF will not go through.
-- hypervisor.sh
hypervisor.sh [/opt/versa/scripts/hyperviser.sh]
[/opt/versa/scripts/hyperviser.sh] gets
gets triggered
triggered after
after executing
executing above
above command.
command.

I got below error while enabling hypervisor:

admin@versa-flexvnf-cli> request system hypervisor enable no-confirm

Error: status " Installing libvirt-20180925-220254-7f58f33-20.1.0.bin "
status " Error installing libvirt-20180925-220254-7f58f33-20.1.0.bin "
[error][2019-01-25 19:48:19] Personal observation
admin@versa-flexvnf-cli>

[admin@versa-flexvnf: ~] $ sudo vim /var/log/versa/extras.log

1. 2019-01-25 19:48:17.687 INFO Installing libvirt-20180925-220254-7f58f33-200.1.0.bin
2 =~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
3
4 >> df: '/var/lib/ureadahead/debugfs/tracing': No such file or directory
5 >> uCPE feature requires at least KB of disk
6 2019-01-25 19:50:53.044 INFO Installing libvirt-20180925-220254-7f58f33-200
.1.0.bin
822 >> Errors were encountered while processing:
823 >> libpixman-1-dev
824 >> libspice-server-dev
825 2019-01-25 20:39:03.428 INFO Installing libvirt-20190125-102558-06ea83a-200

Puru(QA) provided a new image and also need to execute 2 additional linux commands:
versa-flexvnf-20190125-102558-06ea83a-20.1.1-wsm
sudo apt-get update
sudo apt-get –f install

6 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Verification: Before enabling & Post enabling Hypervisor
admin@versa-flexvnf-cli> show system details admin@versa-flexvnf-cli> show system details

Software Details Software Details

Software Release 20.1.1 Software Release 20.1.1
Package name versa-flexvnf-20190125-102558-06ea83a-20.1.1-wsm Package name versa-flexvnf-20190125-102558-06ea83a-20.1.1-wsm

Hardware Details Hardware Details

Hypervisor Type baremetal Hypervisor Type baremetal
Manufacturer Caswell Manufacturer Caswell
SKU Number Versa220-NW SKU Number Versa220-NW
Model CAD-0263 Model CAD-0263
Serial number 1841BA0250 Serial number 1841BA0250
Hardware ID number 1841BA0250 Hardware ID number 1841BA0250
CPU model Intel(R) Atom(TM) CPU C3758 @ 2.20GHz CPU model Intel(R) Atom(TM) CPU C3758 @ 2.20GHz
Number of CPUs 8 Number of CPUs 8
Number of NICs 8 Number of NICs 8
Memory 15.64GiB Memory 15.71GiB
Disk size 117G Disk size 117G
Free NICs 0 Free NICs 0
Free memory 9.83GiB Free memory 9.86GiB
Free disk 107G Free disk 107G
SSD present yes SSD present yes
uCPE Platform no uCPE Platform yes

[ok][2019-01-25 20:41:26] [ok][2019-01-25 13:05:45]

admin@versa-flexvnf-cli>

7 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Verification: OVS Bridges existence/Default Vlan association

OVS bridges created to support connectivity with 3rd Party NFV/Firewall

[admin@versa-flexvnf: ~] $ sudo ovs-vsctl list-br
UCPE-MGMT1
UCPE-MGMT2
UCPE-PORT1
UCPE-PORT2
UCPE-PORT3
UCPE-PORT4
UCPE-PORT5
UCPE-PORT6
UCPE-PORT7
UCPE-PORT8

Default VLAN associated with above bridges: 0

[admin@versa-flexvnf: ~] $ sudo ovs-vsctl br-to-vlan UCPE-MGMT1
0
[admin@versa-flexvnf: ~] $ sudo ovs-vsctl br-to-vlan UCPE-PORT8
0
[admin@versa-flexvnf: ~] $

8 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Negative event: Caswell Issue (Raised a PR # 37241)

Post enabling hypervisor, all uCPE based VNI interfaces came up properly. As a part of negative event, I powered down the v120 hardware and powered it back post which all Ucpe
interfaces got removed from “/var/run/vinterfaces”.

1. “sudo ovs-vsctl list-br/sudo ovs-vsctl show” command outputs were showing proper UCPE bridges and respective ports

2. “/etc/network/interfaces” did have the configuration related to UCPE ports

3. “/var/run/vinterfaces” did not have UCPE interfaces configuration:

1 8:35:71:ec:e1:83 0000:07:00.1 eth0 3 ixgbe

2 8:35:71:ec:e1:7c 0000:05:00.0 eth1 3 igb
3 8:35:71:ec:e1:7d 0000:05:00.1 eth2 3 igb
4 8:35:71:ec:e1:7e 0000:05:00.2 eth3 3 igb
5 8:35:71:ec:e1:7f 0000:05:00.3 eth4 3 igb
6 8:35:71:ec:e1:80 0000:06:00.0 eth5 4 ixgbe
7 8:35:71:ec:e1:81 0000:06:00.1 eth6 4 ixgbe
8 8:35:71:ec:e1:82 0000:07:00.0 eth7 3 ixgbe
9 2:aa:bb:cc:dd:1 0000:00:00.0 wwanusb0 0 af_packet
10 2:aa:bb:cc:dd:3 0000:00:00.0 wwanusb2 0 af_packet
11 2:aa:bb:cc:dd:4 0000:00:00.0 wwanusb3 0 af_packet
~

9 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Negative event: Caswell Issue Resolution

Was able to resolve the issue by rebooting the hardware.

“sudo vim /var/run/vinterfaces” showing all UCPE interfaces!

1 8:35:71:ec:e1:83 0000:07:00.1 eth0 3 ixgbe

2 8:35:71:ec:e1:7c 0000:05:00.0 eth1 3 igb
3 8:35:71:ec:e1:7d 0000:05:00.1 eth2 3 igb
4 8:35:71:ec:e1:7e 0000:05:00.2 eth3 3 igb
5 8:35:71:ec:e1:7f 0000:05:00.3 eth4 3 igb
6 8:35:71:ec:e1:80 0000:06:00.0 eth5 4 ixgbe
7 8:35:71:ec:e1:81 0000:06:00.1 eth6 4 ixgbe
8 8:35:71:ec:e1:82 0000:07:00.0 eth7 3 ixgbe
9 22:83:85:20:2e:4a 0000:00:00.0 UCPE-MGMT1 0 af_packet
10 76:f7:c1:60:50:43 0000:00:00.0 UCPE-MGMT2 0 af_packet
11 56:48:4f:53:54:0 0000:00:00.0 UCPE-PORT1 0 vhost
12 56:48:4f:53:54:1 0000:00:00.0 UCPE-PORT2 0 vhost
13 56:48:4f:53:54:2 0000:00:00.0 UCPE-PORT3 0 vhost
14 56:48:4f:53:54:3 0000:00:00.0 UCPE-PORT4 0 vhost
15 56:48:4f:53:54:4 0000:00:00.0 UCPE-PORT5 0 vhost
16 56:48:4f:53:54:5 0000:00:00.0 UCPE-PORT6 0 vhost
17 56:48:4f:53:54:6 0000:00:00.0 UCPE-PORT7 0 vhost
18 56:48:4f:53:54:7 0000:00:00.0 UCPE-PORT8 0 vhost
19 2:aa:bb:cc:dd:1 0000:00:00.0 wwanusb0 0 af_packet
20 2:aa:bb:cc:dd:2 0000:00:00.0 wwanusb1 0 af_packet
21 2:aa:bb:cc:dd:3 0000:00:00.0 wwanusb2 0 af_packet
22 2:aa:bb:cc:dd:4 0000:00:00.0 wwanusb3 0 af_packet

10 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps to onboard uCPE via VD GUI: Create Vendor Catelog
1. Go to Administration  Inventory  Vendor Catalog

Given below we are trying to create Vendor Catalog from Predefined Vendor and Product type.
Provide Description & Version number of the NFV. “Version” is any used defined value

Location of 3rd Party Firewalls::

http://cloud-images.versa-networks.com/

11 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps to onboard uCPE via VD GUI: Create Service Chain

2a. Go to Workflows  Templates  Service Chains  Create new SC by clicking + sign.

Drag & Drop Fortinet image into SC, Provide appropriate values in the fields.

12 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps to onboard uCPE via VD GUI: Create Service Chain(contd.)

2b. Default mode of operation for 3rd Party FW is Layer2 . Click on 3rd Party VNF to change the mode of operation as Layer 3. You can also
change other parameters such as CPU/Memory. Once all changes are made, click @ “Create”

13 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps to onboard uCPE via VD GUI: Create Workflow Template

3a. Go to Workflows  Templates  Create new template

Please create a workflow template just like any other normal template. No special parameter need to be selected.

14 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps to onboard uCPE via VD GUI: Create Workflow Template (contd)

3b. Go to Workflows  Templates  Create new template

In Interfaces section I had selected “DHCP” option as my CPE was connected with Home internet gateway and will be getting DHCP based ip from it.

15 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps to onboard uCPE via VD GUI: Create Device group & associate SC

4a. Before creating Device template, please create DG and associate it with SC (this is applicable in 20.1 going forward)

In 20.1 Service chain association is moved to Device group level and same hold true in 20.2. Please click at “Edit post-staging” icon and associate SC as shown below.

16 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps to onboard uCPE via VD GUI: Create Device template and provide bind data
5. Create a device template and provide bind data information for Ucpe based interfaces by selecting the Service template and deploy it:

Provided bind data variables will be configured @ FlexVNF end. You need to configure 3 rd party FW interfaces/ip’s separately

In 20.1 release there is no

Save button. Config will be
auto saved after click
deploy button

17 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps need to take before initiating ZTP PROCESS 1: System level knob required
1. If you are going to deploy Palo Alto with default:
Memory: 6gig PLEASE READ CAREFULLY
CPU: 2

Please configure below command in FLEXVNF CLI:

“set system service-options host-hugepage-size 8”

By configuring above knob you are allocating 6gig Memory to Palo Alto VM and
2gig to FlexVNF host. Post configuring this command system reboot is required:

admin@PA-uCPE-Marriott-cli(config)% set system service-options host-hugepage-size 8

[ok][2019-02-06 12:09:02]
[edit]
admin@PA-uCPE-Marriott-cli(config)% commit
Commit complete.
Message from VMOD at 2019-02-06 12:09:05...
Grub parameter have changed.Please reboot the system for changes to take effect....
EOF

Verification post reboot: sudo vim /etc/default/grub.d/50-versa-grub.cfg

GRUB_CMDLINE_LINUX="hugepagesz=1GB hugepages=8 default_hugepagesz=1GB biosdee
vname=0 intel_iommu=on iommu=pt"
Please note different hardware shows different output in above file. For this lab we tested it with Caswell v220 16gig memory.
Default hugepage size of Caswell v220 was 4 which in our case got changed it 8

2. If you are going to deploy Fortigate VM with:

Memory: 2gig
CPU: 2

And if your hardware default hugepages is 4. You do not need to give above command. As FlexVNF will automatically reserve 2GIG for
Fortigate and 2GIG for FlexVNF.

Conclusion: It’s a n + 2 formula which you need to adhere here while configuring above command. 2gig is reserved for FLEXVNF.

18 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Steps need to take before initiating ZTP PROCESS 2: Manually upload the image on uCPE hardware
1. This step is equally more important and need to be followed if you do not want to wait for 2 days waiting on your AWS VD to transfer 2gig of Palo Alto/vSRXV
image(We are assuming that if you have uCPE hardware with you, you will be bootstrapping it from AWS based VD) PLEASE READ CAREFULLY

2. Please go back to “Administration”  “Vendor Catolog” and copy the file name as shown in below snapshot(This is VD auto-generated file name even
though we had uploaded the file from the laptop with this original name: “PA-VM-KVM-8.1.0.qcow2”

3. Transfer the “PA-VM-KVM-8.1.0.qcow2” image (which will be around 2gig in size) to flexvnf /home/versa/images path

4. Rename “PA-VM-KVM-8.1.0.qcow2” with this “3ce31ff9-64b7-4b13-ba75-3b5449696d98_PA-VM-KVM-8.1.0.qcow2”

By executing above steps you will be preventing VD to upload 2gig size of VM image over AWS Cloud to your CPE (which is wastage of time).

Post this step you are good to perform ZTP!!!

19 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Post ZTP: How to access Palo Alto via FlexVNF

[admin@ PA-uCPE-Marriott : /] $ sudo virsh list --all

Id Name State
Palo Alto Credentials:
----------------------------------------------------
User: admin
1 Marriott-uCPE-PA running Password: admin
[admin@PA-uCPE-Marriott: images] $ sudo virsh console 1
Connected to domain Marriott-uCPE-PA
Escape character is ^]

PA-VM login: admin

Password:
Last login: Wed Feb 6 08:16:33 from 10.0.1.113

Number of failed attempts since last successful login: 0 Fortigate Credentials:

User: admin
admin@PA-VM> Password:

One can start/destroyed the VM by using below virsh basic commands:

sudo virsh start Marriott-uCPE-PA (VNF will be in “shutoff” state, so please use this command to bring it on)
sudo virsh destroy Marriott-uCPE-PA

20 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Scalability:
 Total number of 3rd Party VNF’s that can be deployed on single host is dependent on:

 Number of Cores & Memory size of the Host.

 Default Ucpe interface configuration which is from vni-0/300 to 309 can support total 4 VNF’s/3rd Party FW’s:

 VNI-0/300-301 are dedicated for Management interfaces and shared by all 3rd Party FWs. Riverbed utilizes 2 management interfaces. Palo
Alto utilizes just 1

 Remaining VNI-0/302-309 can be divided among VNFs and having each VNF use 2 interfaces for datapath.

21 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
Post ZTP: How to access Palo Alto via VD
1. Access the respective uCPE & click @ “Guest VNF’s”

2. Click “Connect” against SSH and new shell window will open up

Accessing 3rd Party VNF over Web http/https is broken: Bug 34682
CPU/Memory information is not shown correctly: Bug 36642

22 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
“Show interface brief” output post onboarding the device
admin@PA-uCPE-Marriott-cli> show interfaces brief | tab
NAME MAC OPER ADMIN TENANT VRF IP
----------------------------------------------------------------------------------------------
eth-0/0 08:35:71:ec:e1:83 up up 0 global 192.168.2.7/24
ptvi10 n/a up up 2 Marriott-Control-VR 172.29.44.3/32
ptvi11 n/a up up 2 Marriott-Control-VR 172.29.44.1/32
tvi-0/10 n/a up up - -
tvi-0/10.0 n/a up up 2 Marriott-Control-VR 172.29.40.101/32
tvi-0/11 n/a up up - -
tvi-0/11.0 n/a up up 2 Marriott-Control-VR 172.29.44.101/32
tvi-0/2602 n/a up up - -
tvi-0/2602.0 n/a up up 2 PK-INET-Transport-VR 169.254.7.210/31
tvi-0/2603 n/a up up - -
tvi-0/2603.0 n/a up up 2 global 169.254.7.211/31
tvi-0/602 n/a up up - -
tvi-0/602.0 n/a up up 2 PK-INET-Transport-VR 169.254.0.2/31
tvi-0/603 n/a up up - -
tvi-0/603.0 n/a up up 2 Marriott-LAN-VR 169.254.0.3/31
vni-0/0 08:35:71:ec:e1:7c up up - -
vni-0/0.0 08:35:71:ec:e1:7c up up 2 PK-INET-Transport-VR 192.168.2.6/24
vni-0/1 08:35:71:ec:e1:7d up up - -
vni-0/1.0 08:35:71:ec:e1:7d up up 2 Marriott-LAN-VR 172.16.191.1/24
……
vni-0/300 22:83:85:20:2e:4a up up - -
vni-0/300.0 22:83:85:20:2e:4a up up 2 Marriott-Control-VR 172.16.30.1/24
vni-0/301 76:f7:c1:60:50:43 up up - -
vni-0/301.0 76:f7:c1:60:50:43 up up 2 Marriott-Control-VR
vni-0/302 56:48:4f:53:54:00 up up - -
vni-0/302.0 56:48:4f:53:54:00 up up 2 Marriott-LAN-VR 172.16.10.1/24 ----Connected with TRUST LAN Interface of Palo Alto
vni-0/303 56:48:4f:53:54:01 up up - -
vni-0/303.0 56:48:4f:53:54:01 up up 2 Marriott-LAN-VR 172.16.20.1/24 ----Connected with UNTRUST LAN Interface of Palo Alto
………

[ok][2019-02-06 12:41:00]
admin@PA-uCPE-Marriott-cli>

23 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
 Vni0/0.0

Internet-Transport-VR

tvi0/602.0 W-ST-Marriott-LAN-VR-PK-INET

DIA TRAFFIC PATH

Marriott-Control-VR/MP-BGP/Tunnels

mpls-vpn-core-instance

tvi0/603.0 L-ST-Marriott-LAN-VR-PK-INET
Palo Alto VM

172.16.20.2 Untrust
VRF: Marriott-LAN-VR Vni-0/303.0 172.16.20.1

Vni-0/302.0 172.16.10.1
172.16.10.2 Trust

Vni0/1.0 Lan: 172.16.191.1/24

172.16.191.2/24

Logical representation of 3rd party VNF( Management interface is excluded)

24 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential
 Thank You

25 © 2017 Versa and/or its affiliates. All rights reserved. Versa Networks Confidential