Building Data Centre

Networks with VXLAN

BGP-EVPN
Lukas Krattiger– Principal Technical Marketing Engineer
Session Objectives
• Focus on Data Centre Networks and
Fabrics with Overlays
• Closer Look on Packet Encapsulation
(VXLAN)
• Encapsulation and Forwarding
• Underlay – the Transport for the Overlay

• Closer Look on Packet Encapsulation

(BGP EVPN)
• Control-Plane – Exchanging Information
• Optimising the Forwarding
Session Non-Objectives
• Deep-Dive into FabricPath
• There are many Sessions and
Recordings
• Comparison between different
Orchestration and Management
Tools
• Automation Workflows or Services
Catalogs
Albert Einstein
Agenda

• Introduction to Data Centre Fabrics

• VXLAN with BGP EVPN
• Overview
• Underlay
• Control & Data Plane
• Multi-Tenancy

• “Stories” and Use-Cases

• Fabric Management & Automation
Introduction to Data
Centre Fabrics
 Data Centre “Fabric” Journey (Standalone)
STP
VPC
FabricPath
VXLAN

MAN/WA
N

FabricPath VXLAN
/BGP /EVPN

MAN/WA MAN/WA
N N
Data Centre Fabric Properties
Extended Namespace
Scalable Layer-2 Domains
Integrated Route and Bridge
Multi-Tenancy

Hybrid Overlays
Inter-Pod connectivity
Overlay Based Data Centre Fabrics
Desirable Attributes:
RR RR

• Mobility
• Segmentation
• Scale
• Automated & Programmable
• Abstracted consumption models
• Full Cross Sectional Bandwidth
• Layer-2 + Layer-3 Connectivity
• Physical + Virtual
 Overlay Based Data Centre: Edge Devices
Network Overlays Host Overlays Hybrid Overlays

V
V
V
V

V
V

• Router/Switch end-points • Virtual end-points only • Physical and Virtual

• Protocols for Resiliency/Loops • Single admin domain • Resiliency + Scale
• Traditional VPNs • VXLAN, NVGRE, STT • X-Organisations/Federation
• VXLAN, OTV, VPLS, LISP, FP • Open Standards
Data Centre Fabric Properties
• Any subnet, anywhere, rapidly
RR RR
• Reduced Failure Domains
• Extensible Scale & Resiliency
• Profile Controlled Configuration

 Full Bi-Sectional Bandwidth (N Spines)

 Any/All Leaf Distributed Default Gateways

 Any/All Subnets on Any Leaf

Spine/Leaf Topologies
• High Bi-Sectional Bandwidth
• Wide ECMP: Unicast or Multicast
• Uniform Reachability, Deterministic
Latency
• High Redundancy: Node/Link
Failure
• Line rate, low latency, for all traffic
Variety of Fabric Sizes
More Spine, More Bandwidth, More Resiliency
• Fabric size: Hundreds to 10s of
Thousands of 10G ports
• Variety of Building Blocks:
• Varying Size
• Varying Capacity
• Desired oversubscription
• Modular and Fixed

• Scale Out Architecture

• Add compute, service, external
connectivity as the demand grows
VXLAN with BGP EVPN
Agenda

• Introduction to Data Centre Fabrics

• VXLAN with BGP EVPN
• Overview
• Underlay
• Control & Data Plane
• Multi-Tenancy

• “Stories” and Use-Cases

• Fabric Management & Automation
 Overview
Classic Ethernet IEEE 802.1Q
Classic Ethernet
Frame Format Frame
DMAC SMAC 802.1Q Etype Payload CRC

• Traditionally VLAN is expressed

over 12 bits (802.1Q tag) Destination MAC (DMAC)
• Limits the maximum number of Source MAC (SMAC)
segments in a Data Centre to 4096 TPID TCI

VLANs 4 bytes 802.1Q 0x8100

(16 bits)
PCP CFI VID
(3 bits) (1 bits) (12 bits)

Ether Type (Etype)

Data (Payload)

CRC/FCS

VLAN ID
12 bits

TPID = Tag Protocol Identifier, TCI = Tag Control Information, PCP = Priority Code Point,
CFI = Canonical Format Indicator, VID = VLAN Identifier
Overview
Introducing VXLAN
• Traditionally VLAN is expressed
over 12 bits (802.1Q tag)
• Limits the maximum number of
segments in a Data Centre to 4096
VLANs
• VXLAN leverages the VNI field with Classical Ethernet Frame DMAC SMAC 802.1Q Etype Payload CRC

a total address space of 24 bits 50 bytes Original CE Frame

VXLAN Outer IP UDP VxLAN 802.1Q CRC

• Support of ~16M segments MAC
Frame (14) (20) (8) (8)
DMAC SMAC
optional
Etype Payload
(new)

Cisco DFA
Frame
•
8 bits 24 bits 24 bits 8 bits
The VXLAN Network Identifier ags Reserved VNIVNI Reserved

(VNI/VNID) is part of the VXLAN

Header
VNI
 Next-Hop MAC Address

Src VTEP MAC Address

VXLAN Frame Format Dest. MAC Address

Src. MAC Address

48

48
MAC-in-IP Encapsulation VLAN Type 14 Bytes
16
0x8100 (4 Bytes Optional) IP Header
72
Misc. Data
VLAN ID
16
Outer MAC Header Tag
Protocol 0x11 (UDP) 8
Ether Type
16

Underlay
0x0800 Header
16 20 Bytes
Checksum

Source IP 32
50 (54) Bytes of Overhead

Outer IP Header Src and Dst

addresses of the
Source Dest. IP 32
16 VTEPs
Port
UDP Header VXLAN Port 16 Hash of the inner L2/L3/L4 headers of
8 Bytes the original frame.
UDP Length 16 Enables entropy for ECMP Load
VXLAN Header UDP 4789 balancing in the Network.
Checksum 0x0000 16

Overlay

VXLAN Flags
8
RRRRIRRR
Original Layer-2 Frame Allows for 16M
possible
Reserved 24
Segments
8 Bytes
VNI 24

Reserved 8
Data Centre Fabric Properties
 Extended Namespace
Scalable Layer-2 Domains
Integrated Route and Bridge
Multi-Tenancy
Understanding Overlay Technologies
Overlay Services
• Layer 2 Underlay Transport
Tunnel Encapsulation
• Layer 3 Network
• Layer 2 and Layer 3

Control Plane Data Plane

• Overlay Layer 2/Layer 3 Unicast traffic
• Peer Discovery mechanism
• Overlay Broadcast, Unknown Unicast,
• Route Learning and Distribution
Multicast traffic (BUM traffic) forwarding
– Local Learning
– Ingress Replication
– Remote Learning
– Multicast
Why VXLAN?

VXLAN provides a Network with

Segmentation, IP Mobility, and Scale
• “Standards” based Overlay (RFC 7348)
• Leverages Layer-3 ECMP – all links forwarding
• Increased Name-Space to 16M identifier
• Integration of Physical and Virtual
• It’s SDN 
VXLAN Taxonomy (1)

IP Interface Edge Device

Edge Device
Local LAN
Edge Device Segment

IP Interface Edge Device

Edge Device

Edge Device Physical Servers

Local LAN Virtual Servers

Segment
VXLAN Taxonomy (2)

VTEP

VTEP
Local LAN
VTEP Segment

VTEP

VTEP

VTEP Physical Servers

VTEP: VXLAN Tunnel End-Point

Local LAN Virtual Servers VNI/VNID: VXLAN Network Identifier
Segment
Getting the Puzzle Together!
Driving
Standards based
Overlay-
Evolution with
VXLAN BGP
EVPN
What is VXLAN with BGP EVPN?

• Standards based Overlay (VXLAN) with Standards

based Control-Plane (BGP)
• Layer-2 MAC and Layer-3 IP information distribution by
Control-Plane (BGP)
• Forwarding decision based on Control-Plane
(minimises flooding)
• Integrated Routing/Bridging (IRB) for Optimised
Forwarding in the Overlay
• Multi-Tenancy At Scale
 EVPN – Ethernet VPN

Control-
EVPN MP-BGP - RFC 7432
Plane

Multi-Protocol Label Switching Provider Backbone Bridges Network Virtualisation Overlay

Data-
(MPLS) (PBB) (NVO)
Plane
draft-ietf-l2vpn-evpn draft-ietf-l2vpn-pbb-evpn draft-ietf-bess-evpn-overlay

 EVPN over NVO Tunnels (ie VXLAN) for Data Centre

Fabric encapsulations
 Provides Layer-2 and Layer-3 Overlays over simple IP
Networks
Cisco’s VXLAN related IETF RFCs & Drafts

ID Title Category

RFC 7348 Virtual eXtensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualisation Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix- IP Prefix Advertisement in E-VPN Control Plane

advertisement
draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane
Getting the Puzzle Together!
Optimised Networks with VXLAN
Underlay

Overlay
Integrated (VXLAN)
Route/Bridge

BGP
(EVPN)
Agenda

• Introduction to Data Centre Fabrics

• VXLAN with BGP EVPN
• Overview
• Underlay
• Control & Data Plane
• Multi-Tenancy

• “Stories” and Use-Cases

• Fabric Management & Automation
Deployment Considerations
• MTU and Overlays
• Unicast Routing Protocol and IP
Addressing
• Multicast for BUM* Traffic
Replication

*BUM: Broadcast, Unknown Unicast & Multicast

MTU and VXLAN
• VXLAN adds 50 Bytes (or 54 Bytes)
Outer MAC Header to the Original Ethernet Frame

Underlay
• Avoid Fragmentation by adjusting
the IP Networks MTU
50 (54) Bytes of Overhead

Outer IP Header

• Data Centres often require Jumbo

UDP Header
MTU; most Server NIC do support
VXLAN Header
up to 9000 Bytes
• Using a MTU of 9216* Bytes
Overlay
accommodates VXLAN Overhead
Original Layer-2 Frame plus Server max. MTU

*Cisco Nexus 5600/6000 switches only support 9192 Byte for Layer-3 Traffic
Building your IP Network – Interface Principles (1)
• Know your IP addressing and IP Rendezvous-Point Loopback
10.254.254.1
scale requirements Routing Loopback
10.10.10.203/32

• Separate VTEP from Routing p2p Links

Protocol from RP* Loopback 10.1.1.2/30

V
• Best to use individual Aggregates
for the Underlay V
• Unicast Routing p2p** Links p2p Links V
10.1.1.1/30
• Unicast Routing Loopbacks V
• VTEP (NVE) Loopback V p2p Agg: 10.1.1.0/24
• Multicast Routing Loopback (RP) RID Agg: 10.10.10.0/24
Routing Loopback V VTEP Agg: 10.200.200.0/24
10.10.10.101/32
RP Agg: 10.254.254.0/24
• IPv4 only (today) VTEP Loopback
10.200.200.101/32
*RP: Rendezvous-Point (Multicast)
**p2p: Point-to-Point
Building your IP Network – Interface Principles (2)
• Routed Ports/Interfaces
• Layer-3 Interfaces between Spine and
Leaf (no switchport)
• For each Point-2-Point (P2P)
connection, minimum /31 required
V
• Alternative, use IP Unnumbered (/32)

• Use Loopback as Source-Interface V

V
for VTEP (NVE*) V
V
V

*NVE: Network Virtualisation Edge

VTEP: VXLAN Tunnel End-Point
Building your IP Network – Some Math

Example from depicted topology:

4 Spine * 6 Leaf = 24 Point-2-Point (P2P) Links
24 Links * 2 (/31) + 10 RID* + 6 VTEP + 4 Spine

= 48 IP Addresses for P2P Links

= 20 IP Addresses for Loopback Interfaces
V

68 IP Addresses required == /25 Prefix V

V
A More Realistic Scenario: V
4 Spine * 40 Leaf = 160 Point-2-Point (P2P) Link
160 Links * 4 (/30) + 44 RID* + 80 VTEP + 4 Spine V
V
= 640 IP Addresses for P2P Links
= 128 IP Addresses for Loopback Interface

768 IP Addresses required == /22 Prefix *RID: Router ID; Unicast Routing Loopback
IP Unnumbered– Simplifying the Math

Example from depicted topology:

4 Spine + 6 Leaf = 10 Individual Devices

= 6 IP Addresses for Loopback Interface (Used for VTEP)

= 10 IP Address Loopback Interface (RID* & IP Unnumbered)
V
16 IP Addresses required == /28 Prefix
V
V
A More Realistic Scenario: V
4 Spine + 40 Leaf = 44 Individual Devices
V
= 40 IP Addresses for Loopback Interface (Used for VTEP) V
= 44 IP Addresses for Loopback Interface (RID* & IP Unnumbered)

84 IP Addresses required == /25 Prefix

*RID: Router ID; Unicast Routing Loopback
Building your IP Network – Routing Protocols; OSPF
• OSPF – watch your Network type!
• Network Type Point-2-Point (P2P)
• Preferred (only LSA type-1)
• No DR/BDR election
• Suits well for routed interfaces/ports
(optimal from a LSA Database V
perspective)
• Full SPF calculation on Link Change V
• Network Type Broadcast V
• Suboptimal from a LSA Database V
perspective (LSA type-1 & 2) V
• DR/BDR election V
• Additional election and Database
Overhead
Building your IP Network – Routing Protocols; IS-IS
• IS-IS – what was this CLNS?
• Independent of IP (CLNS)
• Well suited for routed interfaces/ports
• No SPF calculation on Link change;
only if Topology changes V
• Fast Re-convergence
• Not everyone is familiar with it V
V
V
V
V

*CLNS: Connection-Less Network Service

Building your IP Network – Routing Protocols; eBGP
• eBGP – Service Provider style
• Two Different Models
• Two-AS
• Multi-AS
• BGP is a Distance Vector
V
• AS* are used to calculate the Path
(AS_Path)
V
• If Underlay is eBGP, your Overlay
V
becomes eBGP
V
V
V

*AS: Autonomous System

Building your IP Network – Routing Protocols; eBGP
• eBGP – TWO-AS, yes it works!
• Total of 8 eBGP Peering (with 4
Spine)
• eBGP peering for Underlay-Routing based
on physical interface
AS#65500
• 4 Spines = 4 BGP Peering per Leaf V
• Advertise all Infrastructure Loopbacks
• eBGP peering for Overlay-Routing V
(EVPN) V
• Loopback to Loopback Peering V
• 4 Spines = 4 BGP Peering V
• Requires some BGP config knobs V
• Disable BGP AS-Path check
• Next-Hop needs to be Unchanged
• Retain all Routes on Spine (not a RR)
Building your IP Network – Routing Protocols; eBGP
• eBGP – Multi-AS
• Total of 8 eBGP Peering (with 4
Spine)
• eBGP peering for Underlay-Routing based
on physical interface
AS#65500
• 4 Spines = 4 BGP Peering per Leaf V
• Advertise all Infrastructure Loopbacks
• eBGP peering for Overlay-Routing V
(EVPN) V
• Loopback to Loopback Peering V
• 4 Spines = 4 BGP Peering V
• Requires some BGP config knobs V
• Next-Hop needs to be Unchanged
• Retain all Routes on Spine (not a RR)
 Multicast Enabled Underlay
May use PIM-ASM or PIM-BiDir (Different hardware has different capabilities)
ASR 1000
Nexus 1000v Nexus 3000 Nexus 5600 Nexus 7000/F3 Nexus 9000 ASR 9000
CSR 1000

Multicast Mode IGMP v2/v3 PIM ASM PIM BiDir PIM ASM / PIM BiDir PIM ASM PIM BiDir PIM ASM / PIM BiDir

• Spine and Aggregation Switches make good Rendezvous-Point (RP) Locations in

Topologies
• Reserve a range of Multicast Groups (Destination Groups/DGroups) to service the Overlay
and optimise for diverse VNIs
• In Spine/Leaf topologies with lean Spine
• Use multiple Rendezvous-Point across the multiple Spines
• Map different VNIs to different Rendezvous-Point for simple load balancing measure
• Use Redundant Rendezvous-Pint
• Design a Multicast Underlay for a Network Overlay, Host VTEPs will leverage this Network
Multicast Enabled Underlay – PIM ASM*
• PIM Sparse-Mode (ASM)
• Redundant Rendezvous-Point using RP
PIM Anycast-RP or MSDP
• Source-Tree or Unidirectional V
Shared-Tree (Source-Tree shown)
• Shared-Tree will always use RP for V
forwarding V
V
• 1 Source-Tree per Multicast-Group V
per VTEP (each VTEP is Source & VTEP1 (S,G) Tree
V
VTEP2 (S,G) Tree
Receiver)
RP Rendezvous-Point
*ASM: Any-Source Multicast
Multicast Enabled Underlay – BiDir-PIM*
• Bidirectional PIM (BiDir)
• Redundant Rendezvous-Point using RP
Phantom-RP
• Building Bi-Directional Shared-Tree V
• Uses shortest path between Source
and Receiver with RP as routing- V
vector V
V
• 1 Shared-Tree per Multicast-Group V
V
VTEPs (*,G) Tree
RP Rendezvous-Point
*BiDir-PIM: Bidirectional PIM
To Remember - Multicast Enabled Underlay

• Multi-Destination Traffic (Broadcast, Unknown Unicast, etc.) needs to be

replicated to ALL VTEPs serving a given VNI
• Each VTEP is Multicast Source & Receiver
• For a given VNI, all VTEPs act as a Sender and a Receiver
• Head-End Replication will depend on hardware scale/capability
• Resilient, efficient, and scalable Multicast Forwarding is highly desirable
• Choose the right Multicast Routing Protocol for your need (type/mode)
• Use redundant Multicast Rendezvous Points (Spine/Aggregation generally preferred)
• 99% percent of Overlay problems are in the Underlay (OTV experience)
Agenda

• Introduction to Data Centre Fabrics

• VXLAN with BGP EVPN
• Overview
• Underlay
• Control & Data Plane
• Multi-Tenancy

• “Stories” and Use-Cases

• Fabric Management & Automation
 Multiprotocol BGP (MP-BGP) Primer
• Multiprotocol BGP (MP-BGP)
• Extension to Border Gateway
RR RR
Protocol (BGP) - RFC 4760
• VPN Address-Family:
• Allows different types of address
families (e.g. VPNv4, VPNv6, L2VPN V2
EVPN, MVPN) V1
• Information transported across single
BGP peering
RR BGP Route-Reflector

V3 iBGP Peering*

*eBGP supported without BGP Route-Reflector

Multiprotocol BGP (MP-BGP) Primer
• VPN segmentation for tenant routing VRF Info
(Multi-Tenancy) VRF Info Name: VRF-A
Name: VRF-A RD: 15:10.0.0.2 (auto)
• Route Distinguisher (RD) RD: Imp
3:10.0.0.1 (auto) RRRoute-Target
RR 65500:50000 (auto)
Imp Route-Target 65500:50000 Exp Route-Target 65500:50000 (auto)
• 8-byte field of VRF parameters VRF Info (auto)
Exp Route-Target 65500:50000
Name: (auto)VRF-A
• value to make VPN prefix unique: RD: 62:10.0.0.3 (auto)
• RD + VPN prefix Imp Route-Target 65500:50000 (auto)
Exp Route-Target 65500:50000 (auto)

V1 V2

RR BGP Route-Reflector

V3 iBGP Peering
Multiprotocol BGP (MP-BGP) Primer
• Cisco’s VXLAN/EVPN does provide VRF Info
automated Route Distinguisher (RD) VRF Info Name: VRF-A
Name: VRF-A RD: 15:10.0.0.2 (auto)
• Automatic uses Type 1 format RD: Imp
3:10.0.0.1 (auto) RRRoute-Target
RR 65500:50000 (auto)
Imp Route-Target 65500:50000 Exp Route-Target 65500:50000 (auto)
• 4-byte IP Address (Router ID) VRF Info (auto)
Exp Route-Target 65500:50000
Name: (auto)VRF-A
• 4-byte Value (VRF ID) RD: 62:10.0.0.3 (auto)
Imp Route-Target 65500:50000 (auto)
vrf context VRF-A Exp Route-Target 65500:50000 (auto)
vni 50000 V2
rd auto V1
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto RR BGP Route-Reflector
route-target both auto evpn V3 iBGP Peering
Multiprotocol BGP (MP-BGP) Primer BGP Advertisement
VPN-EVPN: RD:[MAC_A][IP_A]
• VPN Segmentation for tenant BGP Next-Hop: V1
Route Target: 65500:50000
routing (Multi-Tenancy) Label (L3VNI): 50000
RR RR
• Selective distribute VPN routes -
Route Target (RT)
• 8-byte field of VRF parameter
MAC_A / IP_A >>
• unique value to define the LOCAL
Route-Type2
import/export rules for VPN prefix V1 V2
MAC_A / IP_A >> V1
Route-Type2

RR BGP Route-Reflector

V3 iBGP Peering

Host A
MAC_A / IP_A
Multiprotocol BGP (MP-BGP) Primer BGP Advertisement
VPN-EVPN: RD:[MAC_A][IP_A]
• Cisco’s VXLAN/EVPN does provide BGP Next-Hop: V1
Route Target: 65500:50000
automated Route Target (RT) Label (L3VNI): 50000
• 8-byte Route Target (2 x 4-byte) RR RR
• ASN : VNI

MAC_A / IP_A >>

vrf context VRF-A LOCAL
vni 50000 Route-Type2
V2
rd auto V1
MAC_A / IP_A >> V1
address-family ipv4 unicast Route-Type2
route-target both auto
route-target both auto evpn
address-family ipv6 unicast
route-target both auto RR BGP Route-Reflector
route-target both auto evpn V3 iBGP Peering

Host A
MAC_A / IP_A
 Overlay with Optimised Routing
EVPN Control Plane -- Host and Subnet Route Distribution
BGP Update
RR RR • Host-MAC
Spine • Host-IP
Border • Internal IP Subnet
• External Prefixes
V

V
V
V
V
V
BGP Adjacencies
Route-Reflectors deployed
RR
for scaling purposes (iBGP)
Host Advertisement
Route MAC, IP L2VNI L3VNI NH Encap Seq
(“VLAN”) (“VRF”)
• Host Attaches Type

2 MAC_A, IP_A 30001 50001 IP_V1 8:VXLAN 0

• Host “A” attaches to Edge Device
(VTEP) RR RR

• VTEP V1 advertises Host “A”

reachability information
• MAC and L2VNI [mandatory]
• IP and L3VNI [optional] V1 V2
• depending on ARP

• Additional route attributes

advertised RR BGP Route-Reflector
• MPLS Label1 (L2VNI) V3 iBGP Peering*
• MPLS Label2 (L3VNI) Host A
MAC_A / IP_A
• Extended Communities
 Route Type: Ethernet Segment Ethernet Tag MAC Address
MAC Address IP Address Length IP Address
2 - MAC/IP Identifier Identifier Length

V2# show bgp l2vpn evpn 192.168.1.73

BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.1:32868
BGP routing table entry for
[2]:[0]:[0]:[48]:[0050.56a3.c2bb]:[32]:[192.168.1.73]/272,
version 4
Paths: (1 available, best #1)
Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1
L3VNI Path type: internal, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
L2VNI 10.0.0.1 (metric 3) from 10.0.0.111 (10.0.0.111)
Origin IGP, MED not set, localpref 100, weight 0
Received label 30001 50001
Extcommunity: RT:65501:30001 RT:65501:50001 ENCAP:8 Router MAC:5087.89d4.5495
Originator: 10.0.0.1 Cluster list: 10.0.0.111
Remote VTEP Route Target: Route Target: Overlay Encapsulation: Router MAC of
IP Address L2VNI (VLAN) L3VNI (VRF) 8 - VXLAN Remote VTEP
 Protocol Learning & Distribution
RR RR
MAC, IP L2VNI L3VNI NH MAC, IP L2VNI L3VNI NH

MAC_A, IP_A 30001 50001 local MAC_B, IP_B 30001 50002 local
1
1
1 V2
V1

MAC, IP L2VNI L3VNI NH

MAC_C, IP_C 30001 50001 local

MAC_Y, IP_Y 30002 50001 local

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B

VTEPs advertise End-Host reachability

1 information (MAC,IP) within MP-BGP
Virtual Switch

Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
 Protocol Learning & Distribution
RR RR
MAC, IP L2VNI L3VNI NH MAC, IP L2VNI L3VNI NH

MAC_A, IP_A 30001 50001 local MAC_B, IP_B 30001 50001 local
2
2
2 V2
V1

MAC, IP L2VNI L3VNI NH

MAC_C, IP_C 30001 50001 local

MAC_Y, IP_Y 30002 50001 local

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B

BGP Route-Reflector “reflects” Overlay related

2 reachability information to other VTEPs
Virtual Switch

Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
 Protocol Learning & Distribution
3 3
RR RR
MAC, IP L2VNI L3VNI NH MAC, IP L2VNI L3VNI NH

MAC_A, IP_A 30001 50001 local MAC_B, IP_B 30001 50001 local

MAC_B, IP_B 30001 50001 IP_V2 MAC_A, IP_A 30001 50001 IP_V1

MAC_C, IP_C 30001 50001 IP_V3 MAC_C, IP_C 30001 50001 IP_V3
2
VMAC_Y, IP_Y 30002 50001 IP_V3
MAC_Y, IP_Y 30002 50001
V1IP_V3
3
MAC, IP L2VNI L3VNI NH

MAC_C, IP_C 30001 50001 local

MAC_Y, IP_Y 30002 50001 local

Host A
MAC_A / IP_A V3 MAC_A, IP_A 30001 50001 IP_V1
Host B
MAC_B / IP_B
MAC_B, IP_B 30001 50001 IP_V2
VTEPs receive respective reachability information
3 and installs them related to route-policy into RIB/FIB
Virtual Switch

Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
Subnet Route Advertisement
Route MAC, IP L3VNI NH Encap
(“VRF”)
• IP Prefix Redistribution Type

5 Subnet_A/24 50001 IP_V1 8:VXLAN

• From “Direct” (connected), “Static” or
dynamically learned Routes RR RR

• VTEP V1 advertises local Subnet

through redistribution of “Direct”
(connected) routes
• IP Prefix, IP Prefix Length, and L3VNI V1 V2

• Additional route attributes

advertised
• MPLS Label (L3VNI) RR BGP Route-Reflector
• Extended Communities V3 iBGP Peering*
Subnet Route Advertisement
Route MAC, IP L3VNI NH Encap
(“VRF”)
• If multiple VTEP announce same IP Type

5 Subnet_A/24 50000
50001 IP_V1 8:VXLAN
Prefix, Equal Cost Multipath (ECMP)
5 Subnet_A/24 50001 IP_V2 8:VXLAN
will apply 5 Subnet_A/24 RR
50001 RR IP_V3 8:VXLAN

• VTEP V1 advertises local Subnet

through redistribution of “Direct”
(connected) routes
• IP Prefix, IP Prefix Length, and L3VNI V1 V2

• Additional route attributes

advertised
• MPLS Label (L3VNI) RR BGP Route-Reflector
• Extended Communities V3 iBGP Peering*
Subnet Route Advertisement
Route MAC, IP L3VNI NH Encap
(“VRF”)
• IP Prefix Learning Type

5 Subnet_X/24 50001 IP_V1 8:VXLAN

• via BGP with VRF-Lite (Inter-AS
Option A) RR RR
• via LISP on Nexus 7000/7700
• via other routing protocol (static or
dynamic)
• VTEP V1 participated in external V1 V2
Peering (LISP, BGP, OSPF etc.)
and advertises learned IP Prefixes
into the Fabric
• IP Prefix RR BGP Route-Reflector

• IP Prefix Length V3 iBGP Peering*

• L3VNI
 Route Type: Ethernet Segment Ethernet Tag
IP Prefix Length IP Prefix GW IP Address
5 – IP Prefix Identifier Identifier

V2# show bgp l2vpn evpn 192.168.2.0

BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.0.0.1:3
BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.2.0]:[0.0.0.0]/224, version 3
Paths: (1 available, best #1)
Flags: (0x000002) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
L3VNI 10.0.0.1 (metric 3) from 10.0.0.111 (10.0.0.111)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 50001
Extcommunity: RT:65501:50001 ENCAP:8 Router MAC:5087.89d4.5495
Originator: 10.0.0.1 Cluster list: 10.0.0.111

Remote VTEP Route Target: Overlay Encapsulation: Router MAC of

IP Address L3VNI (VLAN) 8 - VXLAN Remote VTEP
 ARP Suppression
VXLAN/EVPN
RR RR
MAC, IP VNI NH MAC, IP VNI NH

MAC_B, IP_B 30001 IP_V2 MAC_A, IP_A 30001 IP_V1

MAC_C, IP_C 30001 IP_V3 MAC_C, IP_C 30001 IP_V3

MAC_Y, IP_Y 30002 IP_V3 MAC_Y, IP_Y 30002 IP_V3

ARP Request for IP_B
Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF V1 V2
1
ARP Response for IP_B
2 Src MAC: MAC_B
MAC, IP VNI NH
Dst MAC: MAC_A

MAC_A, IP_A 30001 IP_V1

MAC_B, IP_B 30001 IP_V2

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B

1 ARP Request sent for IP_B sent from Host A

Virtual Switch

V1 knows about IP_B and can respond.

2
No need for ARP forwarding across the Network Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
 ARP Handling on Lookup “Miss” (1)
VXLAN/EVPN
Missing RR RR
MAC, IP VNI NH
MAC, IP “B”VNI NH

MAC_C, IP_C 30001 IP_V3 MAC_A, IP_A 30001 IP_V1

MAC_Y, IP_Y 30002 IP_V3 2 MAC_C, IP_C 30001 IP_V3

MAC_Y, IP_Y 30002 IP_V3
ARP Request for IP_B
Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF V1 V2 ARP Request for IP_B
Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF

1 2
MAC, IP VNI NH
MAC_A, IP_A 30001 IP_V1

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B
ARP Request for IP_B
Src MAC: MAC_A
Dst MAC: FF:FF:FF:FF:FF:FF
1 ARP Request sent for IP_B sent from Host A
Virtual Switch

Miss of IP_B. Forward ARP Request to all

2
Ports except source-port (ARP snooping) Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
 ARP Handling on Lookup “Miss” (2)
VXLAN/EVPN
RR RR
MAC, IP VNI NH
MAC, IP VNI NH MAC_A, IP_A 30001 IP_V1
MAC_C, IP_C 30000
30001 V3
IP_V3 MAC_C, IP_C 30001 IP_V3
4
MAC_Y, IP_Y 30001
30002 V3
IP_V3 MAC_Y, IP_Y 30002 IP_V3
ARP Response for IP_B
Src MAC: MAC_B
Dst MAC: MAC_A
MAC_B, IP_B
V1
30001 IP_V2 V2 ARP Response from IP_B
Src MAC: MAC_B
Dst MAC: MAC_A

4
3
MAC, IP VNI NH
MAC_A, IP_A 30000
30001 V1
IP_V1

MAC_B, IP_B 30001 IP_V2

Host A
MAC_A / IP_A V3 Host B
MAC_B / IP_B

3 ARP Response is sent to V2

Virtual Switch

V2 will populate this information in the

4
control-plane (learn) and forward it subsequently Host C Host Y
MAC_C / IP_C MAC_Y / IP_Y
Packet Forwarding (Bridge)
VXLAN/EVPN
RR RR
MAC, IP VNI NH MAC, IP VNI NH

MAC_A, IP_A 30001 Local MAC_B, IP_B 30001 Local

MAC_B, IP_B 30001 IP_V2 MAC_A, IP_A 30001 IP_V1

SMAC: MAC_A SMAC: MAC_A
DMAC: MAC_B DMAC: MAC_B
SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_B DIP: IP_B
1
2 3 4
SMAC: MAC_V1 SMAC: hop-by-hop

Underlay
DMAC: hop-by-hop DMAC: MAC_V2

Underlay
SIP: IP_V1 SIP: IP_V1
DIP: IP_V2 DIP: IP_V2
Host A
MAC_A / IP_A
UDP V3 UDP Host B
MAC_B / IP_B
VXLAN VNID: 30001 VXLAN VNID: 30001

Overlay
Overlay

SMAC: MAC_A SMAC: MAC_A

DMAC: MAC_B DMAC: MAC_B

SIP: IP_A SIP: IP_A

DIP: IP_B DIP: IP_B
Packet Forwarding (Route)
VXLAN/EVPN
RR RR
MAC, IP VNI NH VRF MAC, IP VNI NH VRF

MAC_A, IP_A 30001 Local 50001 MAC_A, IP_A 30001 Local 50001

MAC_F, IP_F 30005 IP_V2 50001 MAC_F, IP_F 30005 E1/4 50001
SMAC: MAC_A SMAC: MAC_GW
DMAC: MAC_GW DMAC: MAC_F
SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_F DIP: IP_F
1
2 3 4
SMAC: MAC_V1 SMAC: hop-by-hop

Underlay
DMAC: hop-by-hop DMAC: MAC_V2

Underlay
SIP: IP_V1 SIP: IP_V1
DIP: IP_V2 DIP: IP_V2
Host A
MAC_A / IP_A
UDP V3 UDP Host F
MAC_F, IP_F
VXLAN VNID: 50001 VXLAN VNID: 50001

Overlay
Overlay

SMAC: MAC_A SMAC: MAC_GW

DMAC: MAC_GW DMAC: MAC_F

SIP: IP_A SIP: IP_A

DIP: IP_F DIP: IP_F
Packet Forwarding (Route) – Silent Host
VXLAN/EVPN
RR RR
MAC, IP VNI NH VRF MAC, IP VNI NH VRF

MAC_A, IP_A 30000 Local 50001 MAC_A, IP_A 30000 Local 50001

Subnet F 30005 IP_V2 50001 Subnet F 30005 E1/4 50001

SMAC: MAC_A SMAC: MAC_GW
DMAC: MAC_GW DMAC: MAC_F
SIP: IP_A
V1 V2 SIP: IP_A
DIP: IP_F DIP: IP_F
1
2 3 4
SMAC: MAC_V1 SMAC: hop-by-hop

Underlay
DMAC: hop-by-hop DMAC: MAC_V2

Underlay
SIP: IP_V1 SIP: IP_V1
DIP: IP_V2 DIP: IP_V2
Host A
MAC_A / IP_A
UDP V3 UDP Host F
MAC_F, IP_F
VXLAN VNID: 50001 VXLAN VNID: 50001

Overlay
Overlay

SMAC: MAC_A SMAC: MAC_GW

DMAC: MAC_GW DMAC: MAC_F

SIP: IP_A SIP: IP_A

DIP: IP_F DIP: IP_F
Data Centre Fabric Properties
 Extended Namespace
 Scalable Layer-2 Domains
Integrated Route and Bridge
Multi-Tenancy
Anycast – One-to-Nearest Association

• a network addressing and RR RR

routing methodology
• datagrams sent from a single
sender to the topologically
✖
nearest node
✔
• group of potential receivers, ✖
all identified by the same
destination address
✔
Distributed IP Anycast Gateway
• Distributed Inter-VXLAN Routing at
Access Layer (Leaf) RR RR

• All Leafs share same gateway IP and

MAC Address for a given Subnet
SVI 100

• Gateway is always active SVI 200

• no redundancy protocol, hello SVI 100

exchange etc. SVI 100

SVI 200

SVI 200
SVI 100
• Distributed state - Smaller ARP SVI 200
SVI 100
tables SVI 200
SVI 100
• Only local attached End-Points SVI 200
(Servers)
SVI 100, Gateway IP: 192.168.1.1, Gateway MAC: AG:AG:AG:AG:AG:AG
SVI 200, Gateway IP: 10.10.10.1, Gateway MAC: AG:AG:AG:AG:AG:AG
Distributed IP Anycast Gateway

RR RR
Spine
bridge SVI 100, Gateway IP: 192.168.1.1
SVI 200, Gateway IP: 10.10.10.1
route V
route
SVI 100
Host3
V MAC: CC:CC:CC:CC:CC:CC
V SVI 200 IP: 192.168.1.33
VLAN 100
V VXLAN VNI 30001

V Host2
MAC: BB:BB:BB:BB:BB:BB
V IP: 10.10.10.22
SVI 100 VLAN 200
Host1 VXLAN VNI 30002
MAC: AA:AA:AA:AA:AA:AA
IP: 192.168.1.11
VLAN 100
VXLAN VNI 30001
Integrated Routing and Bridging (IRB)
VXLAN/EVPN based overlays follow
two slightly different Integrated RR RR

Routing and Bridging (IRB) semantics

• Asymmetric
• Uses an “asymmetric path” from the
Host towards the egressing port of the
VTEP vs. the way back
• Symmetric*
• Uses an “symmetric path” from the
Host towards the egressing port of the
VTEP vs. the way back

*Implemented by Cisco’s VXLAN/EVPN

Consistent Configuration
• Logical Configuration (VLAN, VRF,
VNI) consistently instantiated on ALL RR RR

Leafs
• Optimal for Consistency SVI 100
SVI 200
• Every VLAN/VNI Everywhere
SVI 300
SVI 100
• Sub-Optimal for Scale SVI 100
SVI 200

• Instantiates Resources (VLAN/VNI) SVI 100

SVI 200
SVI 300

even if no End-Point uses it SVI 200

SVI 300
SVI 100
SVI 300
SVI 100 SVI 200

SVI 200 SVI 300

SVI 300
Scoped Configuration
• Logical Configuration (VLAN, VRF,
VNI) scoped to Leafs with respective RR RR

connected End-Points
• Optimal for Scale SVI 200

• Instantiates Resources (VLAN/VNI)

where End-Points are connected SVI 300

SVI 100
• Consistency with End-Points SVI 200
SVI 100
• Configuration Consistency depends SVI 300
SVI 200
on End-Points
SVI 100 SVI 300

SVI 200
Asymmetric IRB
• Similar to todays Inter-VLAN routing
RR RR
• Requires to follow a consistent
configuration of VLAN and L2VNI
across all Switches
• Post routed traffic will leverage
destination Layer 2 Segment

SVI 200

SVI 300
(L2VNI), same as for bridged traffic

SVI 300

SVI 200
 SVI 300
✖
RR

SVI 200
RR

SVI 200
SVI 300
Asymmetric IRB
Asymmetric IRB
L2VNI 30001
L2VNI 30002

Leaf

V V
SVI 300 SVI 200 SVI 200 SVI 300

Host1 Host2 Host3 Host4

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC MAC: DD:DD:DD:DD:DD:DD
IP: 192.168.1.11 IP: 10.10.10.22 IP: 10.10.10.33 IP: 192.168.1.44
VLAN 300 VLAN 200 VLAN 200 VLAN 300
VXLAN VNI 30003 VXLAN VNI 30002 VXLAN VNI 30002 VXLAN VNI 30003
Symmetric IRB
• Similar to Transit Routing Segments
RR RR
• Scoped Configuration of
VLAN/L2VNI; only required where
End-Points (Server) reside
• New VNI (L3VNI) introduced per
virtual routing and forwarding (VRF)

SVI 200

SVI 300
context
• Routed traffic uses transit VNI
(L3VNI), while bridged traffic uses

SVI 300

SVI 200
L2VNI
 SVI 300
RR

SVI 200
RR

SVI 200
SVI 300
Symmetric IRB
Symmetric IRB
L3VNI 50001 (VRF)

Leaf

V V
SVI 300 SVI 200 SVI 200 SVI 300

Host1 Host2 Host3 Host4

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC MAC: DD:DD:DD:DD:DD:DD
IP: 192.168.1.11 IP: 10.10.10.22 IP: 10.10.10.33 IP: 192.168.1.44
VLAN 300 VLAN 200 VLAN 200 VLAN 300
VXLAN VNI 30003 VXLAN VNI 30002 VXLAN VNI 30002 VXLAN VNI 30003
Data Centre Fabric Properties
 Extended Namespace
 Scalable Layer-2 Domains
 Integrated Route and Bridge
Multi-Tenancy
Agenda

• Introduction to Data Centre Fabrics

• VXLAN with BGP EVPN
• Overview
• Underlay
• Control & Data Plane
• Multi-Tenancy

• “Stories” and Use-Cases

• Fabric Management & Automation
What is Multi-Tenancy

• A mode of operation, where multiple independent instances (tenant)

operate in a shared environment.

• Each instance (i.e. VRF/VLAN) is logically isolated, but physically

integrated.
Where can we apply Multi-Tenancy
Multi-Tenancy at Layer-2 Multi-Tenancy at Layer-3
• Per-Switch VLAN-to-VNI mapping • VRF-to-VNI mapping

• Per-Port VLAN Significance • MP-BGP for scaling with VPNs

Layer-2 Multi-Tenancy

RR RR
Spine
bridge

V
VLAN 100
Host3
V MAC: CC:CC:CC:CC:CC:CC
V IP: 192.168.1.33
VLAN 100
V VXLAN VNI 30001

V
V
VLAN 100
Host1
MAC: AA:AA:AA:AA:AA:AA
IP: 192.168.1.11
VLAN 100
VXLAN VNI 30001
Layer-2 Multi-Tenancy – Bridge Domains
VXLAN Overlay
(VNI 30001)

Leaf

V
VLAN 100
Bridge Domain V
VLAN 100

Host1 Host3
MAC: AA:AA:AA:AA:AA:AA MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.33
VLAN 100 VLAN 100
VXLAN VNI 30001 VXLAN VNI 30001
Layer-2 Multi-Tenancy – Bridge Domains
VXLAN Overlay
The Bridge Domain
(VNI is the Layer-2 Segment from Host to Host
30001)

In VXLAN, the Bridge Domain consists of three Components

Leaf

V
VLAN 100
Bridge Domain
1) The Ethernet Segment (VLAN), between Host
V and Switch
VLAN 100
2) The Hardware Resources (Bridge Domain) within the Switch

3) The VXLAN Segment (VNI) between Switch and Switch

Host1 Host3
MAC: AA:AA:AA:AA:AA:AA MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.33
VLAN 100 VLAN 100
VXLAN VNI 30001 VXLAN VNI 30001
VLAN-to-VNI Mapping
VXLAN Overlay
(VNI 30001)

Leaf

V V
VLAN 100 VLAN 100

Host1 Host2 Host3

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.22 IP: 192.168.1.33
VLAN 100 VLAN 100 VLAN 100
VXLAN VNI 30001 VXLAN VNI 30001 VXLAN VNI 30001
CLI Modes - VLAN based (per-Switch)
Leaf#1 • VLAN to VNI configuration on a per-
vlan 100 switch basis
vn-segment 30001
• VLAN becomes “Switch Local
Leaf#2 Identifier”
vlan 100 • VNI becomes “Network Global
vn-segment 30001 Identifier”
Per-Switch VLAN-to-VNI Mapping
VXLAN Overlay
(VNI 30001)

Leaf

V V
VLAN 100 VLAN 200

Host1 Host2 Host3

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.22 IP: 192.168.1.33
VLAN 100 VLAN 100 VLAN 200
VXLAN VNI 30001 VXLAN VNI 30001 VXLAN VNI 30001
CLI Modes - VLAN Based (per-Switch)
Leaf#1 • VLAN to VNI configuration on a per-
vlan 100 switch basis
vn-segment 30001
• VLAN becomes “Switch Local
Leaf#2 Identifier”
vlan 200 • VNI becomes “Network Global
vn-segment 30001 Identifier”
• 4k VLAN limitation has been
removed
Per-Port VLAN-to-VNI Mapping
VXLAN Overlay
(VNI 30001)

Leaf

V V
VLAN 100 VLAN 200 VLAN 300

Host1 Host2 Host3

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC
IP: 192.168.1.11 IP: 192.168.1.22 IP: 192.168.1.33
VLAN 100 VLAN 200 VLAN 300
VXLAN VNI 30001 VXLAN VNI 30001 VXLAN VNI 30001
CLI Modes - VLAN Based (per-Port)

Leaf#1
vlan 2500
vn-segment 30001

interface Ethernet 1/8

switchport mode trunk
switchport vlan mapping enable
switchport vlan mapping 100 2500

interface Ethernet 1/9

switchport mode trunk
switchport vlan mapping enable
switchport vlan mapping 200 2500
CLI Modes - Bridge-Domain Based (per-Port)

Leaf#1
bridge-domain 100
member vni 30001

encapsulation profile vni VLAN100-30001

dot1q 100 vni 30001
interface Ethernet 1/8
encapsulation profile vni VLAN200-30001
no switchport
dot1q 200 vni 30001 service instance 1 vni
encapsulation profile VLAN100-30001 default

interface Ethernet 1/9

no switchport
service instance 1 vni
encapsulation profile VLAN200-30001 default
Layer-3 Multi-Tenancy
VRF-A (VNI 50001)
VRF-B (VNI 50002)
RR RR
Spine SVI 100, Gateway IP: 192.168.1.1 (VRF-A)
route SVI 200, Gateway IP: 10.10.10.1 (VRF-B)
SVI 300, Gateway IP: 172.16.1.1 (VRF-B)
route V
SVI 300
Host3
V IP: 172.16.1.33 (VRF-B)
V SVI 200 VLAN 300

V
V Host2
IP: 10.10.10.22 (VRF-B)
V VLAN 200
SVI 100
Host1
IP: 192.168.1.11 (VRF-A)
VLAN 100
Layer-3 Multi-Tenancy – VRF-VNI or L3VNI
VRF-A VRF-B
(VNI 50001) (VNI 50002)

Leaf
Routing Routing
DomainV
SVI 100
V
SVI 200
Domain V
SVI 300

VRF-A VRF-B
Host1 Host2 Host3
IP: 192.168.1.11 (VRF-A) IP: 10.10.10.22 (VRF-B) IP: 172.16.1.33 (VRF-B)
VLAN 100 VLAN 200 VLAN 300
Layer-3 Multi-Tenancy – VRF-VNI or L3VNI
VRF-A VRF-B
The Routing Domain
(VNI 50001) is the VRF owning multiple
(VNI 50002)across multiple Switches
Subnets

Leaf
Routing Routing
In VXLAN EVPN, the Routing Domain consists of three Components

DomainV
SVI 100
1) The
SVI 200
Domain
V Routing Domains (VRF), local
V to the Switch
SVI 300
2) The Routing Domain (L3VNI) between the Switches
VRF-A VRF-B
3) Multi-Protocol BGP with EVPN Address-Family

Host1 Host2 Host3

IP: 192.168.1.11 (VRF-A) IP: 10.10.10.22 (VRF-B) IP: 172.16.1.33 (VRF-B)
VLAN 100 VLAN 200 VLAN 300
Layer-3 Multi-Tenancy – VRF-Lite
VLAN 1001
Ethernet
VLAN 1002

Leaf

V V
SVI 100 SVI 200 SVI 300 SVI 400

Subnet1 Subnet2 Subnet3 Subnet4

IP: 192.168.1.0/24 (VRF-A) IP: 10.10.10.0/24 (VRF-B) IP: 172.16.1.0/24 (VRF-B) IP: 10.44.44.0/24 (VRF-A)
VLAN 100 VLAN 200 VLAN 300 VLAN 400
Layer-3 Multi-Tenancy – VRF-Lite
vrf context VRF-B vrf context VRF-B

interface eth1/10.1002
VLAN 1001 interface eth1/10.1002
encapsulation dot1q 1002 Ethernet encapsulation dot1q 1002
vrf member VRF-B VLAN 1002 vrf member VRF-B
ip address 10.2.2.1/24 ip address 10.2.2.2/24
ip router ospf 100 area 0.0.0.0 ip router ospf 100 area 0.0.0.0

router ospf 100 router ospf 100

Leaf
vrf VRF-B vrf VRF-B
V
vrf context VRF-A vrf context VRF-A
V
SVI 100 SVI 200 SVI 300 SVI 400
interface eth1/10.1001 interface eth1/10.1001
encapsulation dot1q 1001 encapsulation dot1q 1001
vrf member VRF-A vrf member VRF-A
ip address 10.1.1.1/24 ip address 10.1.1.2/24
ip router ospf 100 area 0.0.0.0 ip router ospf 100 area 0.0.0.0

router ospf 100 router ospf 100

vrf VRF-A vrf VRF-A
Subnet1 Subnet2 Subnet3 Subnet4
IP: 192.168.1.0/24 (VRF-A) IP: 10.10.10.0/24 (VRF-B) IP: 172.16.1.0/24 (VRF-B) IP: 10.44.44.0/24 (VRF-A)
VLAN 100 VLAN 200 VLAN 300 VLAN 400
Layer-3 Multi-Tenancy – MPLS L3VPN
VPN Label “Blue”
MPLS
VPN Label “Red”

Leaf

V V
SVI 100 SVI 200 SVI 300 SVI 400

Subnet1 Subnet2 Subnet3 Subnet4

IP: 192.168.1.0/24 (VRF-A) IP: 10.10.10.0/24 (VRF-B) IP: 172.16.1.0/24 (VRF-B) IP: 10.44.44.0/24 (VRF-A)
VLAN 100 VLAN 200 VLAN 300 VLAN 400
Layer-3 Multi-Tenancy – MPLS L3VPN
vrf context VRF-B vrf context VRF-B
rd 1.1.1.1:200 VPN Label “Blue” rd 1.1.1.2:200
address-family ipv4 unicast address-family ipv4 unicast
route-target import 200:200
MPLS route-target import 200:200
route-target export 200:200 VPN Label “Red” route-target export 200:200

vrf context VRF-A vrf context VRF-A

rd 1.1.1.1:100 rd 1.1.1.2:100
address-family ipv4 unicast
Leaf address-family ipv4 unicast
route-target import 100:100 route-target import 100:100
route-target export 100:100
V route-target export 100:100
V
router SVI
bgp100
65500 SVI 200 routerSVI
bgp30065500SVI 400
address-family ipv4 unicast address-family ipv4 unicast
neighbor 1.1.1.2 remote-as 65500 neighbor 1.1.1.1 remote-as 65500
address-family vpnv4 unicast address-family vpnv4 unicast
send-community extended send-community extended
vrf VRF-A vrf VRF-A
address-family ipv4 unicast address-family ipv4 unicast
vrf VRF-B vrf VRF-B
address-family ipv4 unicast address-family ipv4 unicast
Subnet1 Subnet2 Subnet3 Subnet4
IP: 192.168.1.0/24 (VRF-A) IP: 10.10.10.0/24 (VRF-B) IP: 172.16.1.0/24 (VRF-B) IP: 10.44.44.0/24 (VRF-A)
VLAN 100 VLAN 200 VLAN 300 VLAN 400
Layer-3 Multi-Tenancy – VXLAN EVPN
L3VNI 50001
VXLAN
L3VNI 50002

Leaf

V V
SVI 100 SVI 200 SVI 300 SVI 400

Host1 Host2 Host3 Host4

MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB MAC: CC:CC:CC:CC:CC:CC MAC: DD:DD:DD:DD:DD:DD
IP: 192.168.1.11 (VRF-A) IP: 10.10.10.22 (VRF-B) IP: 172.16.1.33 (VRF-B) IP: 10.44.44.44 (VRF-A)
VLAN 100 VLAN 200 VLAN 300 VLAN 400
VXLAN VNI 30001 VXLAN VNI 30002 VXLAN VNI 30003 VXLAN VNI 30004
Layer-3 Multi-Tenancy – VXLAN EVPN
vrf context VRF-B vrf context VRF-B
vni 50002 L3VNI 50001 vni 50002
rd auto rd auto
address-family ipv4 unicast
VXLAN address-family ipv4 unicast
route-target both auto L3VNI 50002 route-target both auto
route-target both auto evpn route-target both auto evpn

vrf context VRF-A vrf context VRF-A

vni 50001
Leaf vni 50001
rd auto rd auto
address-family ipv4 unicast
V address-family ipv4 unicast
V
route-target both auto route-target both auto
route-target
SVI 100 both auto evpn
SVI 200 route-target
SVI 300 both auto evpn
SVI 400

router bgp 65500 router bgp 65500

address-family ipv4 unicast address-family ipv4 unicast
neighbor 1.1.1.2 remote-as 65500 neighbor 1.1.1.1 remote-as 65500
address-family l2vpn evpn address-family l2vpn evpn
send-community extended send-community extended
vrf VRF-A vrf VRF-A
address-family ipv4 unicast address-family ipv4 unicast
Host1 advertise l2vpn Host2
evpn Host3 Host4
advertise l2vpn evpn
vrf VRF-B
MAC: AA:AA:AA:AA:AA:AA MAC: BB:BB:BB:BB:BB:BB vrf VRF-B
MAC: CC:CC:CC:CC:CC:CC MAC: DD:DD:DD:DD:DD:DD
address-family
IP: 192.168.1.11 (VRF-A) ipv4 unicast (VRF-B)
IP: 10.10.10.22 IP: 172.16.1.33address-family
(VRF-B) IP:ipv4 unicast(VRF-A)
10.44.44.44
VLAN 100 advertise l2vpn evpn
VLAN 200 VLAN 300 advertise l2vpn evpn
VLAN 400
VXLAN VNI 30001 VXLAN VNI 30002 VXLAN VNI 30003 VXLAN VNI 30004
Integrated Route & Bridge + Multi-Tenancy
VRF-A (VNI 50001)
RR RR
Spine
bridge SVI 100, Gateway IP: 192.168.1.1 (VRF-A)
L2VNI 30001 SVI 200, Gateway IP: 10.10.10.1 (VRF-A)
route V
L3VNI 50001 SVI 100
Host3
V MAC: CC:CC:CC:CC:CC:CC
V SVI 200 IP: 192.168.1.33
VLAN 100
V VXLAN VNI 30001

V Host2
MAC: BB:BB:BB:BB:BB:BB
V IP: 10.10.10.22
SVI 100 VLAN 200
Host1 VXLAN VNI 30002
MAC: AA:AA:AA:AA:AA:AA
IP: 192.168.1.11
VLAN 100
VXLAN VNI 30001
Integrated Route & Bridge + Multi-Tenancy
VRF-A (VNI 50001)
RR RR
Spine
SVI 100, Gateway IP: 192.168.1.1 (VRF-A)
SVI 200, Gateway IP: 10.10.10.1 (VRF-A)
V
SVI 100
Host3
V MAC: CC:CC:CC:CC:CC:CC
V SVI 200 IP: 192.168.1.33 (VRF-A)
VLAN 100
V VXLAN VNI 30001

V Host2
MAC: BB:BB:BB:BB:BB:BB
V IP: 10.10.10.22 (VRF-A)
SVI 100 VLAN 200
Host1 VXLAN VNI 30002
MAC: AA:AA:AA:AA:AA:AA
IP: 192.168.1.11 (VRF-A)
VLAN 100
VXLAN VNI 30001
Data Centre Fabric Properties
 Extended Namespace
 Scalable Layer-2 Domains
 Integrated Route and Bridge
 Multi-Tenancy
Agenda

• Introduction to Data Centre Fabrics

• VXLAN with BGP EVPN
• Overview
• Underlay
• Control & Data Plane
• Multi-Tenancy

• “Stories” and Use-Cases

• Fabric Management & Automation
“Stories” and Use-Cases
 • Yesterday: VXLAN, yet another
Overlay
VXLAN • Data-Plane only (Multicast based
Flood & Learn)
applicability • Today: VXLAN for the creation of
scalable DC Fabrics – Intra-DC
evolves as the • Control-Plane, active VTEP discovery,

Control Plane Multicast and Unicast (Head-End

Replication)

evolves!
Story #1: Scalable Data Centre Fabric

• VXLAN based Data Centre Fabric

• BGP EVPN Control-Protocol (Overlay)
• OSPF for Underlay Routing (Unicast)
• PIM ASM with Anycast-RP for BUM Replication (Underlay)
• Distributed IP Anycast Gateway
 Story #1: Scalable Data Centre Fabric (1)
Spine

Border Leaf

Leaf

Leaf

Leaf

Leaf

p2p Agg: 10.1.1.0/24 Leaf

RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
Story #2: Scalable Data Centre Fabric

• VXLAN based Data Centre Fabric

• BGP EVPN Control-Protocol (Overlay)
• eBGP for Underlay Routing (Unicast)
• eBGP Multi-AS Design
• Ingress Replication for BUM (Underlay)
• Distributed IP Anycast Gateway
 Story #2: Scalable Data Centre Fabric (1)
Spine AS65500

AS65555

AS65504

AS65503

AS65503

AS65502

p2p Agg: 10.1.1.0/24 AS65501

RID Agg: 10.10.10.0/24
VTEP Agg: 10.200.200.0/24
RP Agg: 10.254.254.0/24
 • Yesterday: VXLAN, yet another
Overlay
VXLAN • Data-Plane only (Multicast based
Flood & Learn)
applicability • Today: VXLAN for the creation of
scalable DC Fabrics – Intra-DC
evolves as the • Control-Plane, active VTEP discovery,

Control Plane Multicast and Unicast (Head-End

Replication)

evolves! • Future: VXLAN for DCI – Inter-DC

• DCI Enhancements (ARP
caching/suppress, Multi-Homing,
Failure Domain isolation, Loop
Protection etc.)
What is the Elephant in the Room?
Note sure if it is an Elephant
VXLAN for Interconnecting Networks
Story #3: Inter-Fabric Connectivity

• Option 1: End-to-End Fabric Stretch

• Option 2: Fabric-DCI-Fabric (2-box)
• Option 3: Fabric-DCI-Fabric L3-DCI (1-box)
• Option 4: Fabric-DCI-Fabric L2-DCI (1-box)
Inter-Fabric Connectivity (Option 1)
• Multiple BGP-EVPN Control-Plane
EVPN Control-Plane Domain 1 Domains
• End-to-End reachability for VTEP
V • End-to-End reachability for BUM
V
Replication
V • Multicast / Ingress Replication
V

V
V
• End-to-End Data-Plane
encapsulation
V
V

VXLAN Encapsulation
EVPN Control-Plane Domain 2
Inter-Fabric Connectivity (Option 2)
• Multiple BGP-EVPN Control-Plane
EVPN Control-Plane Domain 1 Domains
• Normalisation via Ethernet (MPLS,
VRF-lite & IEEE 802.1Q Trunk) at
V
V the Border
V • Separate Data-Plane (DP)
V
DCI
V
encapsulation per Domain
V
• Multicast / Ingress Replication
V
V

VXLAN Encapsulation
EVPN Control-Plane Domain 2 DCI Encapsulation
Inter-Fabric Connectivity (Option 3 / Option 4)
• Multiple BGP-EVPN Control-Plane
EVPN Control-Plane Domain 1 Domains
• Integrated Hand-Off with Data-Plane
separation
V
V • Option 3 – L3 DCI
• L3-LISP, MPLS, EVPN
V
V
• Option 4 – L2 DCI
V • OTV, L2-LISP, EVPN
V
• Separate Data-Plane (DP)
V encapsulation per Domain
V • Multicast / Ingress Replication

VXLAN Encapsulation
EVPN Control-Plane Domain 2 DCI Encapsulation
Inter-Fabric Connectivity
Option 1 Option 2 Option 3/4

Underlay Control Plane Unified Underlay Domain Separated Underlay Domains Separated Underlay Domains

Overlay Control Plane Separated Overlay Control-Plane Domains

Overlay Data Plane Single Data-Plane Separated Data-Planes Separated Data-Planes

Unified Underlay Domain (All

BUM Replication in DCI Dependency on DCI Choice (Unicast/Multicast)
Multicast or All Ingress Replication)

ARP Flood Suppression

yes yes yes
(DCI)

Unknown Unicast Flood

no yes yes
Suppression (DCI)

Broadcast
no yes yes
Suppression/Limit (DCI)

Layer-2 Loop Prevention Loop mitigation (Edge Protection) VPC at Border Loop mitigation (At DCI)
Fabric Management &
Automation
How to Achieve Data Centre Automation
• Simplify
• Do not start with the most difficult task (low hanging Fruits)
• Standardise
• Find common Denominators and create Templates
• Automate repetitive Tasks
• Use Templates for Simple Tasks and use Automation (e.g. create VLAN, SVI, VRF)
• Abstract
• Take a step back and look at the WHOLE
• Cisco ACI
Anatomy of Data Centre Automation
VMM Chef Openstack
Puppet NX-API Ansible
API

Underlay Overlay Hybrid Overlay Inter-Domain

Management Simplified Provisioning
Management & Management and Multi-Fabric
- integration of
- Network Element- - Overlay Services - Seamless LISP
Physical and Virtual
Management (Layer 2/Layer 3) and MPLS
Workload Mobility,VTEPs
Service Agility
-Topology Overview - Service Chaining integration
-Configuration - Optimising Inter-
Deployment Multi-tenancy Domain integration
-Cross DC Mobility

IP Fabric

Network Infrastructure
 Fabric Management & Operations
Element Day-0: Day- 1: Day-2:
management:
Configuration Configuration and Visibility,
Hardware (POAP) Configuration Configuration
Management, Management increments,
Health Status, and Underlay compare changes.
Inventory Management Automated
Configuration

Compute
Integration

Troubleshooting
Device Auto-Configuration (POAP)
Day 0, Day 0.5 and Day 1
1. Easy way to unbox, rack the device, and not enter any base CLI
configuration. Just rack, power, and plug into the management network.
2. Provides a standard and consistent configuration across of the data centre
network devices.
3. Provides a standard and consistent images to deploy to all of the data centre
devices.
Recommended Reading
Using TRILL, FabricPath, and VXLAN:
Designing Massively Scalable Data
Centres (MSDC) with Overlays
• Sanjay K. Hooda
• Shyam Kapadia
• Padmanabhan Krishnan
ISBN-10: 1-58714-393-3

ISBN-13: 978-1-58714-393-9
Recommended Viewing
Cisco Programmable Fabric Using
VXLAN with BGP EVPN LiveLessons
• David Jansen
• Lukas Krattiger
ISBN-10: 0-13-427229-3

ISBN-13: 978-0-13-427229-0
Q&A
Complete Your Online Session Evaluation
Give us your feedback and receive a
Cisco 2016 T-Shirt by completing the
Overall Event Survey and 5 Session
Evaluations.
– Directly from your mobile device on the Cisco Live
Mobile App
– By visiting the Cisco Live Mobile Site
http://showcase.genie-connect.com/ciscolivemelbourne2016/
– Visit any Cisco Live Internet Station located
throughout the venue
Learn online with Cisco Live!
T-Shirts can be collected Friday 11 March Visit us online after the conference
for full access to session videos and
at Registration presentations.
www.CiscoLiveAPAC.com
Thank you