BRKACI-2505-Deploying Kubernetes With Cisco ACI

#CLUS
Deploying Kubernetes with

Cisco ACI
Camillo Rossi – TME DCBU

BRKACI-2505
#CLUS
Session Objectives
• At the end of the session, the participants should be able to:
• Have a general understanding of containers
• Have a general understanding of Kubernetes
• Understand how ACI and Kubernetes integration is deployed
• Initial assumption:
• The audience already has a good knowledge of ACI main concepts
(Tenant, BD, EPG, L2Out, L3Out, etc.)
#CLUS BRKACI-2505 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKACI-2505

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
ACI Kubernetes Integration – Value proposition
• Allow containers direct access to the ACI policy model, so that they
can participate as first-class citizens within an ACI fabric
• Allow seamless integration of containers, VMs, and physical
devices on an ACI fabric
• Support native policy semantics, so that a container application that
is specified using Kubernetes NetworkPolicy will work correctly out
of the box
• I.E. The same config works on Google Cloud, AWS and ACI
• Leverage fabric resources and Opflex to assist accelerating
Kubernetes service load balancing
ACI Kubernetes Integration – Value proposition
• ACI admin can (optionally) define EPGs and contracts that are used
to secure communication from/to/within the Kubernetes cluster
• EPG is selected based on annotation, which can also be used to
dynamically define new EPGs
Agenda
• ACI-Kubernetes value proposition
• Introduction to Containers
• Container Management and Orchestration
• ACI and Kubernetes Solution Overview
• Demos
• Q&A
For your
reference
Linux
fundamentals
Understanding
Containers Origins
Lightweight Process Virtualisation For your
is not new reference
• Create multiple views of the root filesystem to isolate applications

and processes
• Lightweight Process Virtualisation is not a new concept:
• Solaris Zones
• BSD jails
• Linux chroot
• AIX WPARs (Workload Partitions)
For your
What is new? reference
• Linux Kernel 3.8 (February 2013) added support for namespaces

and cgroups
• Namespaces are like chroot for processes, but also applied to
network, UTS (Unix Timesharing), mount, IPC and users (UIDs).
• Example: Network namespaces enable the creation of multiple, isolated
routing tables that operate independently. Multiple hostnames could also
be used:
• #ip netns add myns1 -> Create a new Namespace “myns1”
What is UTS namespaces allow a single system to appear to have
different host and domain names to different processes For your
new? The PID namespace provides processes with an independent
reference
set of process IDs (PIDs) from other namespaces
Namespaces architecture
Mount namespaces
control mount points
IPC namespaces isolate processes from SysV style inter-
For your
What is new?
process communication
reference
Network namespaces virtualise the network stack
Namespaces architecture User namespaces are a feature to

provide both privilege isolation and
user identification segregation
across multiple sets of processes
What is new?
• Cgroups provide resource management capabilities
• Processes can be grouped into user-defined group of tasks, for
optimised system resource usage
• Cgroups move resource allocation from the process level to the
application level by grouping and labeling processes into
hierarchies
• Resource allocation includes CPU time, block IO, RAM and network
bandwidth
What is new?
• Cgroups architecture:
CPU Network Memory Storage I/O
Cgroup1
Cgroup2
What are
containers?
What is a container?
• A container is a binary executable, packaged with dependencies
and intended for execution in a private namespace with optional
resource constraints.
• This provides the containers multiple isolated operating system
environments with their own file system, network, process and
block I/O space on the same host
Compute Virtualisation and Containers
App App App
Bins/ Bins/ Bins/

App App App
VMs Libs Libs Libs
Containers Bins/ Bins/ Bins/
Guest Guest Guest
OS OS OS Libs Libs Libs
Hypervisor Container Engine (Linux)
Host OS Host OS
Server Hardware Server Hardware
Virtualisation Containers
Similarities
• They provide a way to abstract resources

• They define logical boundaries to the resources they consume
• They enable multiple OS instances to run on the same host
• They share the resources of the host
Differences
• Containers can only run the same OS as the host

• Containers share the same kernel as the host
• Containers are faster to provision and boot
• Containers have lower overhead as there is no need for the
hypervisor layer
Containers Current Challenges
• Containers images management
• Orchestration of containers across multiple hosts
• Lack of standards
• Integration with virtualisation and cloud tools
• Networking management  Addressed by ACI
Why Containers?
Application Architectural Evolution
f()
Service Microservice Function
Autonomous Single Purpose Single Action
Loosely-coupled Stateless Event Sourced
Independently Scalable Ephemeral
Automated
An effective platform for micro-services
• Containers are ideal candidates to run micro-services:
• Micro-services define stateless, loosely coupled application components
communicating over API’s, running in different runtime environments.
• Containers meet new application requirements as they provide:
• Density
• Speed
• Portability
• Low overhead management
Best Practices when using containers
• Containers should be ephemeral
• Are immutable
• Don’t store Data within containers
• Don’t build large images
• Don’t run more than one process per container
• Don’t rely on IP addresses
Portability is key
• As containers are portable, they are very relevant in the following
areas:
• Continuous Integration
• Hybrid cloud strategy
• Scale out applications
• Web development
Containers runtimes
• A container runtime enables users to make effective use of
containerisation mechanisms by providing APIs and tooling that
abstract the low level technical details
• LXC - Open Source Solutions (OSS)
• Docker – OSS and commercial
• Rkt – part of CoreOS, OSS and commercial
• VMware Integrated Container (aka Project Bonneville) - Proprietary
• RunC - OSS
• Garden – part of Pivotal Cloud Foundry, OSS and commercial
Docker
Docker provides an integrated
technology suite that enables
development and IT operations
teams to build, ship, and run
distributed applications anywhere.
A Docker containers wrap a piece
of software in a complete
filesystem that contains
everything needed to run: code,
runtime, system tools, system
libraries – anything that can be
installed on a server.
This guarantees that the software
will always run the same,
regardless of its environment.
For your
reference
In version 0.9, Docker switched

from LXC to their own execution
driver, called libcontainer.
Docker consists of two main components:
Containers
Docker Engine
– the actual app
running on the host.
Docker Hub – SaaS component for managing

and sharing containers.
Dockerfile and Registry
• A Dockerfile is simply a text file containing instructions on how to
build a Docker image
• It can add components on top of an existing image
• Images are available online on the Docker hub repository
• Local, private registry can be created.
• A registry is an instance of the registry container image
More about
Docker Images…
Docker Images
• A Docker image is made up of filesystems layered over each other.
Docker Images
• The storage driver is responsible for presenting these layers as a
single, unified file system.
Docker Images
• When you start a container, Docker creates an empty, read-write
layer on top of the stack – all changes are made in this layer.
Docker Images
• Docker uses “copy-on-write” container layers.
• If a file needs to be modified, it is copied into the read-write layer
first.
Docker Images
• This means that multiple containers can share a single copy of the
image.
Docker Networking
Option 1 - None
• Doesn’t create any network interface for the container
Option 2 - Bridge Mode
• Default mode where Docker attaches containers to Docker0 bridge
• Containers in the same host can talk to each other
• Containers on different host can’t talk to each other (or anything
else) easily
Option 2 - Bridge Mode
• Requirement to use iptables to NAT the containers IPs
Kubernetes BD/VRF
eth0 eth0
Host-1 iptables iptables Host-2
docker0 Bridge docker0 Bridge

172.17.42.1 172.17.42.1
veth774786d vethde4e22e veth994786d vethab4e22e
Eth0 Eth0 Same IPs for the Eth0 Eth0

172.17.0.12 172.17.0.13 containers is not a typo 172.17.0.12 172.17.0.13
Container 1 Container 2 Container 3 Container 4
For your
Option 3 - Host Mode reference
• Connect containers to Host network stack

• All the network interfaces defined on the host will be available to
the container (every container will have the same IP address as the
host) Host-1
Eth0 (host interface)
192.168.0.2
Eth0 Eth0
192.168.0.2 192.168.0.2
Container 1 Container 2
For your
Option 4 - Mapped Container Mode reference
• Container is mapped to another container network stack

• Filesystem, processes and other resources are kept separate
• They share network resources (IP, interfaces)
Host-1
docker0 Bridge
172.17.42.1
veth774786d
Eth0 Eth0
172.17.0.12 172.17.0.12
Container 1 Container 2
All these Docker Networking options are complex
• Iptables rules must be created manually to allow/NAT traffic to
containers ports…
• Containers on different hosts can’t communicated to each other
even if they are in the same L2 domain
• Needs to manually manage port-mappings
• Prone to errors
Docker network driver plugins
• Network plugins can be used to extended Docker networking
support to a wide range of networking technologies, such as
VXLAN, IPVLAN, MACVLAN or something completely different.
For your
A tale of two standards… reference
• Custom network driver (CNM) • Container network interface (CNI)

• Proposed by CoreOS
• Proposed by Docker
• Plugin-based
• Plugin-based • Multiple runtime (Docker, LXC etc..)
• Supports Only Docker • Containers con join 1 or more networks
• Containers con join 1 or more networks • Supports namespace isolation
• Supports namespace isolation • Integrates with IPAM
• Simple
• Integrates with IPAM
• Complex
For your
Kubernetes choose… reference
• CNI
• Would you like to know more?
• http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-
libnetwork.html
Containers
Orchestrators
In a multi-host environment, containers need to
• Have network reachability
• Be fault-tolerant
• Easily scalable
• Use resources optimally
• Can discover other containers/application automatically
• Communicate with each other
• Can be update/rollback without any downtime
• Expose services in an easy and reliable way
Container Orchestration basic features
• Bring multiple hosts together and make them part of a cluster
• Schedule containers to run on different hosts
• Help containers running on one host reach out to containers
running on other hosts in the cluster
• Bind containers and storage
• Bind containers of similar type to a higher-level construct, like
services, so we don't have to deal with individual containers
• Keep resource usage in-check, and optimize it when necessary
• Allow secure access to applications running inside containers.
Containers Orchestrators
• Docker Swarm is a Container Orchestrator provided by Docker, Inc.
It is part of Docker Engine.
• Kubernetes started by Google, now part of the Cloud Native
Computing Foundation project.
• Mesos Marathon is one of the frameworks to run containers at
scale on Apache Mesos.
• Amazon EC2 Container Service (ECS) is a hosted service provided
by AWS to run Docker containers at scale on its infrastructure.
• Hashicorp Nomad is the Container Orchestrator provided
by HashiCorp.
Kubernetes
• Kubernetes is an open source Container Orchestration system for
automating deployment, scaling and management of containerised
applications.
• It was inspired by the Google Borg System and with its v1.0 release
in July 2015, Google donated it to the Cloud Native Computing
Foundation (CNCF).
• Generally, Kubernetes has new releases every three months. The
current stable version is 1.11 (as of Jan 2019).
Kubernetes and Docker
• Kubernetes uses Docker to execute/run the containers
• Kubernetes adds, on top of Docker, all the intelligence and features
of an orchestrator
Kubernetes Features
• Automatic binpacking
Kubernetes automatically schedules the containers based on
resource usage and constraints, without sacrificing availability.
• Self-healing
Kubernetes automatically replaces and reschedules the containers
from failed nodes. It also kills and restarts containers which do not
respond to health checks, based on existing rules/policy.
• Horizontal scaling
Kubernetes can automatically scale applications based on resource
usage like CPU and memory. It also supports dynamic scaling
based on customer metrics
Kubernetes Features (cont.)
• Service discovery and Load balancing
Kubernetes groups sets of containers and refers to them via a DNS
name. This DNS name is also called a Kubernetes service.
Kubernetes can discover these services automatically, and load-
balance requests between containers of a given service.
• Automated rollouts and rollbacks
Kubernetes can roll out and roll back new versions/configurations of
an application, without introducing any downtime.
Kubernetes Features (cont.)
• Secrets and configuration management
Kubernetes can manage secrets and configuration details for an
application without re-building the respective images. With secrets,
we can share confidential information to our application without
exposing it to the stack configuration, like on GitHub.
• Storage orchestration
With Kubernetes and its plugins, we can automatically mount local,
external, and storage solutions to the containers in a seamless
manner, based on Software Defined Storage (SDS).
• Batch execution
Besides long running jobs, Kubernetes also supports batch execution.
Kubernetes Architecture
• At a very high level, Kubernetes has the following main
components:
• One or more Master Nodes
• One or more Worker Nodes
• Distributed key-value store, like etcd.
For your
Kubernetes – etcd reference
• Kubernetes uses etcd to store the cluster state. etcd is a

distributed key-value store based on the Raft Consensus
Algorithm. Raft allows a collection of machines to work as
a coherent group that can survive the failures of some of
its members. At any given time, one of the nodes in the
group will be the Master, and the rest of them will be the
Followers. Any node can be treated as a Master.
• etcd is written in the Go programming language. In
Kubernetes, besides storing the cluster state, etcd is also
used to store configuration details such as subnets,
ConfigMaps, Secrets, etc.
Kubernetes Components – Master Node
• The Master Node is responsible for managing the Kubernetes
cluster. Master node access methods are CLI, GUI or APIs.
• For fault tolerance, there can be more than one Master Node.
• To manage the cluster state, Kubernetes uses etcd, and all
Master Nodes connect to it. etcd is a distributed key-value
store. The key-value store can be part of the Master Node. It
can also be configured externally, in which case, the Master
Nodes connect to it.
Kubernetes Components – Worker Node
• A Worker Node is a machine (VM, physical
server, etc.) which runs the containers using
pods and is controlled by the Master Node.
• pods are scheduled on the Worker Nodes
Kubernetes - pod
• A pod is the scheduling unit in Kubernetes. It is a
logical collection of one or more containers which are
always scheduled together.
• The set of containers composed together in a pod
share an IP.
[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system
NAME READY STATUS RESTARTS AGE
aci-containers-controller-1201600828-qsw5g 1/1 Running 1 69d
aci-containers-host-lt9kl 3/3 Running 0 72d
aci-containers-host-xnwkr 3/3 Running 0 58d
aci-containers-openvswitch-0rjbw 1/1 Running 0 58d
aci-containers-openvswitch-7j1h5 1/1 Running 0 72d
Kubernetes – Deployment
• Deployments are a collection of pods providing the same service
• You describe the desired state in a Deployment object, and the
Deployment controller will change the actual state to the desired
state at a controlled rate for you
• For example you can create a deployment that declare you need to
have 2 copies of your front-end pod.
[root@k8s-01-p1 ~]# kubectl get deployment --namespace=kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
aci-containers-controller 1 1 1 1 72d
Kubernetes – Services
• A service tells the rest of the Kubernetes environment (including other
pods and Deployments) what services your application provides.
• While pods come and go, the service IP addresses and ports remain the
same.
• Kubernetes automatically load balance the load across the replicas in the
deployment that you expose through a Service
• Other applications can find your service through Kubernetes service
discovery.
• Every time a service is create a DNS entry is added to kube-dns
[root@k8s-01-p1 ~]# kubectl get svc --namespace=kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 11.96.0.10 <none> 53/UDP,53/TCP 72d
Kubernetes – External Services
• If there are external IPs that route to one or more cluster nodes,
Kubernetes services can be exposed on those external IPs.
• Traffic that ingresses into the cluster with the external IP (as
destination IP), on the service port, will be routed to one of the
service endpoints.
• External IPs are not managed by Kubernetes and are the
responsibility of the cluster administrator.
[root@k8s-01-p1 ~]# kubectl get svc front-end --namespace=guest-book
front-end 11.96.0.33 11.3.0.2 80:30002/TCP 3m
Kubernetes – Ingress
• An Ingress is a collection of rules that allow inbound connections to
reach the cluster services.
• It can be configured to give services externally-reachable URLs,
load balance traffic, terminate SSL, offer name based virtual
hosting, and more
• Think of NGINX
[root@k8s-01-p1 ~]# kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
test-ingress * 80 7s
Kubernetes - Labels
• Kubernetes uses labels as “nametags” to identify things.
• Can be used to indicate roles, stability, or other important
attributes.
• You can query anything in Kubernetes via a label.
• i.e. Return all the pod that are running “PreProduction” workload
[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system -l component=kube-apiserver

NAME READY STATUS RESTARTS AGE
kube-apiserver-k8s-01-p1 1/1 Running 0 72d
Kubernetes - Annotations
• Similar to labels but are NOT used to identify and select object
• Used in ACI, yes soon we will be speaking about ACI and
Kubernetes 
[root@k8s-01-p1 ~]# kubectl describe node k8s-01-p1 | more
Name: k8s-01-p1
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=k8s-01-p1
node-role.kubernetes.io/master=
Annotations: node.alpha.kubernetes.io/ttl=0
opflex.cisco.com/pod-network-ranges={"V4":[{"start":"11.2.0.130","end":"11.2.1.1"}]}
opflex.cisco.com/service-endpoint={"mac":"66:85:9a:e9:ef:2f","ipv4":"11.5.0.3"}
volumes.kubernetes.io/controller-managed-attach-detach=true
Kubernetes – Namespace
• Groups everything together:
• Pod
• Deployment
• Volumes
• Services
• Etc…
All Together: A K8S Cluster
Deployment1 Namespace
pod1
Node1 Container
Application
A node can be part of
Several Namespaces
pod2 Service
Node2 Container 1.1.1.1:80
Application
pod[n]
Node[N] Container
Application
ACI and
Kubernetes
Solution Overview
Application Centric Infrastructure
Any Application – Any hypervisor
• Build to and support open systems

and standards
• Common pervasive gateway and
policy based routing provide optimal
network connectivity
• Policy consistency provides for
containers running reliably and
securely
• Ease of deploying, scaling and
managing
Cisco ACI and Container Integration
ACI and Containers
Unified networking: Containers, VMs, and
bare-metal
Micro-services load balancing integrated in

fabric for HA / performance
Secure multi-tenancy and seamless

integration of Kubernetes network policies
and ACI policies
OpFlex OVS OpFlex OVS
Visibility: Live statistics in APIC per
Node Node container and health metrics
Deploying Kubernetes Clusters to ACI Fabrics
• Kubernetes Clusters are deployed to
an ACI Tenant (existing or new)
EXT
• ACI CNI installer will create Bridge
Domains and EPGs for Kubernetes
node and POD subnets on a given
VRF (in tenant or common) Tenant: AcmeTenant
VRF:
(in tenant or from common)
• The ACI CNI installer will also create APP DB APP DB
the VMM Domain and other relevant
VRF
objects APP DB APP DB
K8s Cluster-01 K8s Cluster-02

• ACI supports multiple clusters per
fabric.
• The ACI CNI plugin is compatible with

Multi-POD
ACI Network Plugin for Kubernetes
Native Security Policy Support
Developer Network Administrator
Infosec
1 1 Fabric Bring Up
1 Build containers
Define Container
Define BDs, Context and
Network Policy
2 AP
2 Deploy/Scale Clusters
` Get VLAN Pools Allocated

3
3 Annotate policy For Each EPG
EPG
Full Infrastructure Visibility,
4 Telemetry
Opflex/OVS
Infrastructure Policy
Enforcement
Host level Policy
Enforcement WEB APP WEB APP DB
Server 1 Server 2
ACI Fabric
ACI VMM Domain for Kubernetes
Network Policy
Kubernetes Technical Description
• Network policies of Kubernetes supported using standard
upstream format but enforced through OpFlex / OVS using
ACI Policies APIC Host Protection Profiles
• Kubernetes app configurations can be moved without
modification to/from ACI and non-ACI environments
• Embedded fabric and virtual switch load balancing
• PBR in fabric for external service load balancing
• OVS used for internal service load balancing
OpFlex OVS OpFlex OVS

• VMM Domain for Kubernetes
• Stats per namespace, deployment, service, pod
Node Node
• Physical to container correlation
ACI CNI Plugin Components
• aci-containers-controller
• Handle IPAM
• Management of endpoint state
• Policy Mapping (annotations)
• Controls Load Balancing
• Pushes configurations into the APIC
• aci-containers-host is a DaemonSet composed of 3 containers:
• mcast-daemon:
• Handles Broadcast, unknown unicast and multicast replication
• aci-containers-host:
• Endpoint metadata
• Pod IP Address management
• Container Interface Configuration
• opflex-agent:
• Support for Stateful Security Groups
• Manage configuration of OVS
• Render policy to openflow rules to program OVS.
• Handles loadbalanced services (connection tracking, natting, etc…)
• aci-containers-openvswitch
• Bridge traffic from containers to physical interfaces
• Enforce policies
ACI and
Kubernetes
Security Model
Support for Network Policy in ACI namespace-a
namespace-b
• Specification of how selections of pods are allowed to
communicate with each other and other network endpoints.
• Network namespace isolation using defined labels
• directional: allowed ingress pod-to-pod traffic
• filters traffic from pods in other projects
• can specify protocol and ports (e.g. tcp/80)
Policy applied to namespace: namespace-a

kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
name: allow-red-to-blue-same-ns
spec:
podSelector:
matchLabels:
type: blue
ingress:
- from:
- podSelector:
matchLabels:
type: red
Mapping Network Policy and EPGs
Cluster Isolation Namespace Isolation Deployment Isolation
Single EPG for entire cluster.

Each namespace is mapped to its own Each deployment mapped to an EPG
(Default behavior) EPG. Contracts tightly control service traffic
No need for any internal contracts. Contracts for inter-namespace traffic.
Key Map EPG NetworkPolicy Contract
Dual level Policy Enforcement by ACI
Native API Default deny all traffic

apiVersion: networking.k8s.io/v1
Both Kubernetes Network Policy and kind: NetworkPolicy
metadata:
ACI Contracts are enforced in the name: default-deny
Linux kernel of every server node

spec: podSelector: {}
policyTypes:
that containers run on.

- Ingress
- Egress
Containers are mapped to EPGs and

contracts between EPGs are also
enforced on all switches in the fabric
where applicable.
Both policy mechanisms can be used in conjunction.
ACI Basic
Configuration
Connecting Bare Metal Nodes to the Fabric
• Virtual Port Channels enable simple and Kubernetes Node 01
fast link redundancy for Kubernetes bare
metal nodes
bond0
• Can use standard based LACP between enp8s0 enp9s0
K8s nodes and leaf pair for optimal load vPC Policy
balancing and link-failure convergence eth1/10 eth1/10
Group
• vPC Policy Group per K8s node server

ACI Leaf1 ACI Leaf2
• AEP per K8s Cluster

• Enable infraVLAN on the AEP. k8s-node-vPC-PolicyGroup AEP-k8s-cluster-01
Make sure InfraVLAN

is enabled for AEP
Connecting VM Nodes to the Fabric
• As of ACI 3.1 it is supported to also run the ACI CNI Kubernetes Node 01
plugin for clusters running on vSphere VMs

ens160
• The VMs should be running on a VMware VMM
Domain. ESXi-node VDS
vPC Policy
• Prior to installation, ensure the infraVLAN is Group
activated for the AEP used for the ESXi hosts. eth1/10 eth1/10
• The VMs will be connected to a PortGroup that will ACI Leaf1 ACI Leaf2
be created by the ACI CNI installer tool.

esxi-node-vPC-PolicyGroup AEP-VDS-ESX-CLUSTER
Make sure InfraVLAN

is enabled for AEP
Kubernetes Nodes will require the following
minimum interfaces
• InfraVLAN – sub-interface over which we build the opflex channel
• Node IP – sub-interface used for the Kubernetes API host IP
address
• (Optional) OOB Management – sub-interface or physical interface
used optionally for OOB access.
IMPORTANT: the default route must be on the Node IP interface.
acc-provision
• ACI Container Controller Provision:
• Takes a YAML file containing the parameters of your configuration
• Generates and pushes most of the ACI config
• Generates Kubernetes ACI CNI containers configuration
acc-provision --flavor=kubernetes-1.13 -a -u admin -p pass –c config.yml –o cni_conf.yml
Used to select if we are deploying

APIC user and Configuration file Output file for ACI CNI
kubernetes 1.6, 1.7 or OpenShift
password config
3.6
acc-provision – configuration file (1)
aci_config:
system_id: KubeSpray # Tenant Name and Controller Domain Name
apic_hosts: # List of APIC hosts to connect for APIC API
- 10.67.185.102
vmm_domain: # Kubernetes VMM domain configuration
encap_type: vxlan # Encap mode: vxlan or vlan
mcast_range: # mcast range for BUM replication
start: 225.22.1.1
end: 225.22.255.255
mcast_fabric: 225.1.2.4
nested_inside: # (OPTIONAL) If running k8s node as VMs specify the VMM Type and Name.
type: vmware # Only vmware for now, ports groups created automatically with system_id name
name: ACI
# The following resources must already exist on the APIC,

# they are used, but not created by the provisioning tool.
aep: ACI_AttEntityP # The AEP for ports/VPCs used by this cluster
vrf: # The VRF can be placed in the same Tenant or in Common.
name: vrf1
tenant: KubeSpray # This can be the system-id or common
l3out:
name: l3out # Used to provision external IPs
external_networks:
- default_extepg # Default Ext EPG, used for PBR redirection
acc-provision – configuration file (2)
#
# Networks used by Kubernetes
#
net_config:
node_subnet: 10.32.0.1/16 # Subnet to use for nodes
pod_subnet: 10.33.0.1/16 # Subnet to use for Kubernetes Pods
extern_dynamic: 10.34.0.1/24 # Subnet to use for dynamic external IPs
extern_static: 10.35.0.1/24 # Subnet to use for static external IPs
node_svc_subnet: 10.36.0.1/24 # Subnet to use for service graph
kubeapi_vlan: 4011 # The VLAN used by for nodes to node API communications
service_vlan: 4013 # The VLAN used by LoadBalancer services
infra_vlan: 3456 # The ACI infra VLAN used to establish the OpFlex tunnel with the leaf
#
# Configuration for container registry
# Update if a custom container registry has been setup
#
registry:
image_prefix: noiro # DO NOT CHANGE
acc-provision acc-provision creates tenant EPGs
for nodes and Pods
Within the tenant selected the
provisioning tool creates a ‘Kubernetes’
Application Profile with three EPGs:
• ‘kube-nodes’: for node interfaces,
mapped to PhysDom
• ‘kube-system’: for system PODs (i.e.
kube-dns), mapped to VMMDom
• ‘kube-default’: base EPG for all
containers on any namespace by
default, mapped to VMMDom
acc-provision creates tenant BDs for nodes and
Pods
• kube-nodes-bd:
• Only used for kube-node EPG
• Maps to node_subnet
• kube-pod-bd:
• Any pod will be assigned an IP from this BD
Subnet
• Used for kube-default, kube-system and any
other user defined POD EPGs.
• Maps to pod_subnet
• Cluster…-service:
• BD for PBR/SG services
• Created when ACI CNI plugin is deployed
acc-provision configures the required tenant
contracts
• The required minimum set of
contracts are automatically
configured to ensure basic cluster
functionality
• DNS
• Health-check
• ICMP
• Kube-API
• Administrator can define additional

contracts if/when required
ACI Fabric Configuration – L4-L7 Devices
• Created once the ACI CNI plugin is deployed
• Dynamically updated if nodes are added or removed from the k8s cluster
• Service Graph Template: Specify a template for PBR redirection
ACI Fabric Configuration – Required L3OUT
• Fabric Administrator must create and
configure the L3OUT that will be used to
expose external services
• The L3OUT name and the Default
Networks name must match those in
acc-provision configuration
• The External EPG used must be
associated with the contract that allows
nodes to reach repos, registry, etc.
The nodes will probably require a
contract to the L3Out (not
provisioned) to reach repos,
registry, etc.
Once all this is done, the cluster is ready for
installing Kubernetes
Kubernetes Node-N
REPOS
Kubernetes Node-02
Kubernetes Master Bond0.4001

Kubernetes Node-01 Bond0.4093 L3Out
11.1.0.101/16 10.1.80.67
Bond0.4001 Bond0.4093
11.1.0.102/16 10.1.80.68 External
Bond0.4001 Bond0.4093 Bond0.4001 Bond0.4093 EPG
11.1.0.100/16 10.1.80.66 11.1.0.101/16 10.1.80.67
kube-node-EPG Contract
(mapped to AcmeTenant-pdom, vlan4001)
kube-node-bd 11.1.0.1/16, 00:22:bd:f8:19:ff
The contract to the External EPG

EPG – access|default on the L3Out is important to
(vlan 4093) ensure access to repositories.
BD default, VRF overlay-1
ACI Fabric Configuration – Container Domain
Fabric admin can search

APIC for k8s nodes, masters,
pods, services …
View pods per node, map to

encapsulation, physical point
APIC keeps inventory of pods in the fabric.
and their metadata (labels,
annotations), deployments,
replicasets, etc.
Demo 1
Deploying an Application
Demo 1 – guestbook application
For your
reference
• The guestbook application uses Redis to store its data. It writes its
data to a Redis master instance and reads data from multiple Redis
slave instances.
• The code can be found at:
https://kubernetes.io/docs/tutorials/stateless-
application/guestbook/
Demo 1 –guestbook application
For your
reference
Create a namespace for our application

cisco@k8s-01:~/demo/guestbook1$ kubectl create namespace guestbook
Deploy all the components

cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook apply -f complete.yaml
deployment "frontend" created
service "frontend" created
deployment "redis-master" created
service "redis-master" created
deployment "redis-slave" created
service "redis-slave" created
Check POD status

cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get pod -o wide
READY STATUS RESTARTS AGE IP NODE
frontend-1768566195-mj43h 1/1 Running 0 2m 10.33.1.11 k8s-02
frontend-1768566195-tpw75 1/1 Running 0 2m 10.33.0.153 k8s-03
frontend-1768566195-vljrh 1/1 Running 0 2m 10.33.0.155 k8s-03
redis-master-2365125485-8hg60 1/1 Running 0 2m 10.33.0.152 k8s-03
redis-slave-3837281623-p4fs7 1/1 Running 0 2m 10.33.1.12 k8s-02
redis-slave-3837281623-qw894 1/1 Running 0 2m 10.33.0.154 k8s-03
Note: All the commands are executed from the Kubernetes master node
For your
Demo 1 – Check APIC Controller Domain reference
Visibility on where this specific

pod is running
APIC has complete visibility into

k8s objects
For your
Demo 1 – Check APIC EPG reference
POD Name
For your
Demo 1 – Cluster Services reference
• By default every POD is exposed only to the k8s cluster via a

Service IP. You can imagine this as a Virtual IP of a load balancer.
• With the ACI CNI plugin the LoadBalancing for the internal cluster
services is performed by OVS
cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get service
frontend 10.37.0.124 <nodes> 80:32677/TCP 25m
redis-master 10.37.0.162 <none> 6379/TCP 25m
redis-slave 10.37.0.136 <none> 6379/TCP 25m
• Try to access the service from one of the nodes

cisco@k8s-01:~/demo/guestbook1$ curl 10.37.0.124
<html ng-app="redis">
<head>
<title>Guestbook</title>
!SNIP!
Demo 2
Placing PODs/Namespaces into EPGs
For your
Demo 2 - APIC Steps reference
• Create an EPG under your application

• BD = your pod BD
• VMM Domain = Your Kubernetes Domain
• Every POD in an EPG needs to be able to communicate with:

• kube-system for cluster wide DNS resolution
• kube-node for health monitoring probes
• Top Tip: Use EPG contract masters and inherit contracts from kube-
default!
For your
Demo 2 - acikubectl reference
• Utility to manage and troubleshoot the k8s cluster

• Can be used to annotate Namespeces or Deployments with the
Tenant/App/EPG names
cisco@k8s-01:~/demo/guestbook1$ acikubectl set default-eg namespace guestbook -t KubeSpray -a kubernetes -g guestbook

Setting default endpoint group:
Endpoint Group:
Tenant: KubeSpray
App profile: kubernetes
Endpoint group: guestbook
For your
Check under your EPG reference
• All your PODs should now have moved from kube-default to

guestbook
Exposing Services
The extern_dynamic subnet
• Defined in acc-provision configuration file
• An IP address will be automatically selected from this subnet to
expose your service outside of the k8s cluster/fabric
• Expose the service as “LoadBalancer” (as per kubernetes standard)
• The extern_dynamic subnet is not associated to a BD: You need to
configure your external router with static routes toward your L3OUT
for this subnet
cisco@k8s-01:~/demo/guestbook1$ kubectl --namespace=guestbook get svc frontend
frontend 10.37.0.124 10.34.0.5 80:32677/TCP 5h
extern_dynamic
Service Graphs and PBR
Every time a service is exposed the ACI CNI controller will deploy:
• An External EPG with a /32 match for the Service IP
• A new contract between the svc_ExtEPG and the default_ExtEPG*
• A Service Graph with PBR redirection containing every node where an exposed
POD is running
Client RTR L3Out
default_ExtEpg Cons Pod1

0.0.0.0/0 OVS Node1
Pod2
Node2 Pod3
Contract PBR Service Graph OVS
Pod4
Svc_x_ExtEPG
OVS
Pod5
10.34.0.5/32
NodeN
Prov NodeN
* defined in the acc-provision config file

Service Graphs and PBR – Packet walk
1. Client send a request to 10.34.0.5, ACI performs Longest Prefix Match (LPM)
on the SIP and classify the traffic in the default_extEPG
2. ACI does a routing lookup for 10.34.0.5, IP does not exist in the fabric, we
should route it out however LPM places it in the Svc_x_ExtEPG
3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to
one of the nodes
SIP DIP
192.168.1.100 10.34.0.5
Client RTR L3Out
default_extEpg Cons Pod1

0.0.0.0/0 OVS Node1
Pod2
Node2 Pod3
Pod4
Svc_x_ExtEPG
OVS
Pod5
10.34.0.5/32
NodeN
Prov NodeN
1. Client send a request to 10.34.0.5, ACI performs Longest Prefix Match (LPM)
on the SIP and classify the traffic in the default_extEPG
2. ACI does a routing lookup for 10.34.0.5, IP does not exist in the fabric, we
should route it out however LPM places it in the Svc_x_ExtEPG
3. PBR redirection is triggered and the traffic is LoadBalanced by the fabric to
one of the nodes
SIP DIP
Client RTR L3Out
192.168.1.100 10.34.0.5

0.0.0.0/0 OVS Node1
Pod2
Node2 Pod3
Pod4
Svc_x_ExtEPG
OVS
Pod5
10.34.0.5/32
NodeN
Prov NodeN
4. The K8S node is not expecting any traffic directed to the external service IP so
OVS will perform NAT as required
5. If there are multiple POD on a single node OVS will perform a second stage
LB to distribute the load between Pods running on the same node
SIP DIP
192.168.1.100 10.34.0.5
SIP DIP
Client RTR L3Out 192.168.1.100 PodX IP

0.0.0.0/0 OVS Node1
Pod2
Node2 Pod3
Contract PBR Service Graphc OVS
Pod4
Svc_x_ExtEPG
OVS
Pod5
10.34.0.5/32
NodeN
Prov NodeN
4. PodX replies to the client
5. OVS restore the original external Service IP
6. PBR redirection is not triggered since the source EPG is the Shadow EPG of
the PBR node
7. Traffic is routed back to the client (and is permitted by the contract)
DIP SIP
Client RTR L3Out
192.168.1.100 PodX IP
DIP SIP
OVS Node1
0.0.0.0/0 192.168.1.100 10.34.0.5 Pod2
Node2 Pod3
Pod4
Svc_x_ExtEPG
OVS
Pod5
10.34.0.5/32
NodeN
Prov NodeN
Demo 3
Exposing Services
Exposing a service
• Simply choose the LoadBalancer ”type” in the service definition
• The ACI CNI plug in will:
• Automatically pick a free IP from the extern_dynamic subnet
• Create the ExtEPG
• Create contracts
• Create PBR redirection rules
• Deploy the service graph
Scalability
• Currently the scalability of exposing external service with PBR is
limited by the number of external EPGs per L3OUT.
• ACI 4.0 supports 250 external EPGs per L3 OUT per leaf *
• This is a soft limit and will increase with time
• But we want more! So?
*For details check:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/4-x/verified-scalability/Cisco-ACI-Verified-Scalability-Guide-401.html
Scaling External
Services with
Ingress
• Composed of two parts:
• Ingress Resources: collection of rules that defines how inbound
connections can reach the internal cluster services.
• Ingress controller: responsible for fulfilling the Ingress, usually with a
virtual loadbalancer (nginx, ha-proxy)
• Ingress controller can be shared between multiple namespaces
• It can be configured to give services externally-reachable URLs,
load balance traffic, terminate SSL, offer name based virtual hosting
etc…
• Easy integration with DNS: configure a wildcard DNS record
(*.camillo.com) pointing to the IP of the ingress controller
Ingress Ingress Resource
Controller (I am app1)
Namespace
app1
Namespace Pod1
ingress Service
Pod2
app1.camillo.com Ingress Cont
Ingress Cont 11
Client
app2.camillo.com
IngressCont
Ingress Cont 22
Namespace
IngressCont
Ingress Cont N
N app2
Pod1
Service
Pod2
Ingress Resource
(I am app2)
ACI and Kubernetes Ingress
• Expose the Ingress Controller via Service Graph with PBR
• A single Service Graph/ExtEPG can now host as many services as
we want
• Ingress Controller can be scaled as needed
• If you create a dedicated EPG for ingress you need the following
contracts:
• All the contracts used in kube-default (remember contract inheritance)
• Consume: Kube-API, Ingress need to be able to speak with the Kube API
server
• Consume: any required ports between Ingress Controller and the service
you wan to expose
ACI and Kubernetes Ingress Ingress Resource
(I am app1)
Ingress
Controller
Client
Namespace
app1
Namespace Pod1
RTR L3Out
ingress Service
default_extEpg
Pod2
Cons
0.0.0.0/0 Ingress Cont
Ingress
Ingress Cont
Cont111
Contract PBR Service Graph IngressCont

Ingress
Ingress Cont 22
Cont 2
Namespace
Ingress_ExtEPG
10.34.0.6/32
IngressCont
Ingress
Ingress Cont N
Cont N app2
Prov
Pod1
Service
Pod2
Ingress Resource
(I am app2)
ACI and Kubernetes External Services -
Summary
• Two options (can be used at the same time even for the same
service)
• Exposing services via ingress
• Exposing up to 250 services directly with Service Graph with PBR
Demo 4
Exposing Services with Ingress
Container to
Non-Container
Communications
Container to Non-Container Communications
• In production environments is preferred, for example, to run
services like high performance databases as VMs or Bare
Metal Servers
• This calls for the ability to easily provide communication
between K8S POD and VMs/Bare Metal
• Simply deploy a contract between your EPGs, ACI will do
the rest!
• This works for any VMM domain and Physical Domains, for
example you can have a Container Domain using VXLAN
speaking with a Microsoft SCVMM Domain using VLAN.
Container to Non-Container Communications:
F5 Integration with ACI CNI
F5 As an Ingress LoadBalancer
• It is possible to use a Physical or Virtual BIG-IP to expose your
services
• Runs a k8s-bigip-ctlr to send config to the BIG-IP
• Two mode of operation:
• LoadBalance the traffic directly between the PODs: the BIG-IP needs
direct connectivity to the POD subnet.
• LoadBalance the traffic to the NodePort of the service that you are
exposing, this is suboptimal as all the nodes are added in the LB pool so
you can send traffic to nodes where no PODs are present and you have
to trombone the traffic around the cluster
ACI – F5 Design Booth 1323
Contract Contract
Clients BIG-IP EPG POD EPG
VLAN-10 VXLAN-123456
BIG-IP LB POD-1 POD-2 POD-n
Demo 5
F5 and ACI CNI
What about
Service Meshes?
(ISTIO)
ISTO is Transparent
• The sidecar proxy sits inside the original application POD and is completely transparent to
our CNI plugin as per the picture below.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 135
Kubernetes Cluster
Node Failure
Kubernetes Cluster Node Failure Detection
• Kubernetes Monitors by default all the node in the clusters
• Depending on the configuration, node failure detection and
container restart can take from ~50s to 5min. This will depend on
your specific configuration.
• Once a node is detected as NotReady (failed) the aci-container-
controller will update the ACI configuration as required i.e. a failed
node will be removed from the PBR redirection policy
ACI CNI redundancy during node failure
aci-containers-host and aci-containers-openvswitch
• DataPlane of the CNI Plugin
• Start and Stop with the Node
• If isolated from the network they will try to reconnect to the leaf
ACI CNI redundancy during node failure
aci-containers-controller (acc)
• Stateless
• Does not sit in the data-path
• In case of failure k8s will restart it on a different node
ACI CNI redundancy
common configuration mistake
aci-containers-controller
• Node connects to OOB and ACI Fabric
• K8S Cluster communications are happening over the ACI Fabric
• aci-containers-controller communicates with APIC via OOB
ACI
Fabric
aci-containers-controller1 Node1 Node2
OOB
ACI CNI redundancy
common configuration mistake
aci-containers-controller
• Node 1 losses connectivity with the ACI Fabric (interface down)
• K8S master will detect node1 as lost and restart acc on Node2
• The old instance of acc1 is still running and will keep injecting the
old config, overwriting the configuration changes pushed by acc2
• When designing your network ensure that acc communication with
the APIC goes trough the fabric ACI
Fabric
aci-containers-controller1 Node1 Node2 aci-containers-controller2
OOB
What is coming
ACI 4.2 Upcoming Features
• POD IP Source NAT
• Handled directly by OVS
• Annotation based, can be enabled at Namespace, Deployment or POD
level.
• Docker EE Support
How can I build my
own lab?
Cisco Container Platform
Native Kubernetes (100% Upstream)
Direct updates and best practices from open source community
Hybrid Cloud Optimized

E.g.: Google, AWS, …
Integrated
Networking | Management | Security | Analytics
Turnkey Solution
For Production-Grade Container Flexible Deployment Model
Environments VM | Bare metal  HX, UCS, ACI | Public cloud
Easy to acquire, deploy and manage | Open and consistent | Extensible platform | World-class advisory and support
Not officially
aci_kubeadm supported
• Set of ansible scrips to deploy an single master cluster using ACI

CNI plugin
• Open Source (not supported by TAC/Cisco etc…)
• Optionally can clone VM templates and configure everything
providing a 1-Click deployment solution for your lab
• https://github.com/camrossi/aci_kubeadm
Yes it is me… Did I
mentioned is not
officially supported? 
Not officially
kubespray_aci supported
• Set of ansible scrips to deploy an Multi-master cluster using ACI

CNI plugin
• Open Source (not supported by TAC/Cisco etc…)
• https://github.com/camrossi/kubespray_aci
Still me, still not

officially supported? 
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Related Session
• DEVNET-2617: ACI CNI integrated with a CI/CD Pipeline
• LABACI-2010: ACI Runs Everything, ACI CNI, SCVMM, VMWare
Thank you
#CLUS
#CLUS

BRKACI-2505-Deploying Kubernetes With Cisco ACI

Загружено:

Сведения о документе

Оригинальное название

Авторское право

Доступные форматы

Поделиться этим документом

Поделиться или встроить документ

Параметры публикации

Этот документ был вам полезен?

Это неприемлемый материал?

Авторское право:

Доступные форматы

BRKACI-2505-Deploying Kubernetes With Cisco ACI

Загружено:

Авторское право:

Доступные форматы

#CLUS

Deploying Kubernetes with

Camillo Rossi – TME DCBU

Webex Teams will be moderated cs.co/ciscolivebot#BRKACI-2505

• Create multiple views of the root filesystem to isolate applications

• Linux Kernel 3.8 (February 2013) added support for namespaces

Namespaces architecture User namespaces are a feature to

CPU Network Memory Storage I/O

App App App

Bins/ Bins/ Bins/

Hypervisor Container Engine (Linux)

Server Hardware Server Hardware

• They provide a way to abstract resources

• Containers can only run the same OS as the host

In version 0.9, Docker switched

Docker Hub – SaaS component for managing

• Doesn’t create any network interface for the container

docker0 Bridge docker0 Bridge

Eth0 Eth0 Same IPs for the Eth0 Eth0

Container 1 Container 2 Container 3 Container 4

• Connect containers to Host network stack

• Container is mapped to another container network stack

• Custom network driver (CNM) • Container network interface (CNI)

• Kubernetes uses etcd to store the cluster state. etcd is a

[root@k8s-01-p1 ~]# kubectl get pod --namespace=kube-system -l component=kube-apiserver

• Build to and support open systems

Micro-services load balancing integrated in

Secure multi-tenancy and seamless

objects APP DB APP DB

K8s Cluster-01 K8s Cluster-02

• The ACI CNI plugin is compatible with

` Get VLAN Pools Allocated

OpFlex OVS OpFlex OVS

Policy applied to namespace: namespace-a

Single EPG for entire cluster.

Key Map EPG NetworkPolicy Contract

Native API Default deny all traffic

Linux kernel of every server node

that containers run on.

Containers are mapped to EPGs and

• vPC Policy Group per K8s node server

• AEP per K8s Cluster

Make sure InfraVLAN

plugin for clusters running on vSphere VMs

be created by the ACI CNI installer tool.

Make sure InfraVLAN

IMPORTANT: the default route must be on the Node IP interface.

Used to select if we are deploying

# The following resources must already exist on the APIC,

• Administrator can define additional

Kubernetes Master Bond0.4001

kube-node-bd 11.1.0.1/16, 00:22:bd:f8:19:ff

The contract to the External EPG

Fabric admin can search

View pods per node, map to

Create a namespace for our application

Deploy all the components

Check POD status

Visibility on where this specific

APIC has complete visibility into

• By default every POD is exposed only to the k8s cluster via a

• Try to access the service from one of the nodes

• Create an EPG under your application