EVPN VXLAN FOR

QFX10K
Prabakaran A
Kumaraguru Radhakrishnan

1 Copyright © 2014 Juniper Networks, Inc.

 Agenda
• EVPN-VXLAN Overview

• EVPN-VXLAN L2GW and L3GW Overview

• EVPN-VXLAN Packet Flow

• ECMP

• ESI

• EVPN-VXLAN Collapsed Topology

• MAC+IP Route

• VMTO

• EVPN-VXLAN Type5

• EVPN-VXLAN Troubleshooting

2 Copyright © 2014 Juniper Networks, Inc.

 Why VXLAN
• Traditional network isolation techniques such as IEEE 802.1Q VLAN provide 4096
LAN segments (through a 12-bit VLAN identifier) and may not provide enough
segments for large cloud deployments.

• Virtual Extensible LAN (VXLAN), defines a 24-bit LAN segment identifier that
provides segmentation at cloud scale

• VXLAN can also enable migration of virtual machines between servers across
Layer 3 networks.

• VXLAN provides an architecture that customers can use to expand their cloud
deployments with repeatable pods in different Layer 2 domains

3 Copyright © 2014 Juniper Networks, Inc.

 VXLAN definitions from draft
• VXLAN is a Layer 2 overlay scheme over a Layer 3 network.
• Each overlay is termed a VXLAN segment.
• Only VMs/servers within the same VXLAN segment can communicate
with each other; directly.
• Each VXLAN segment is identified through a 24 bit segment ID,
termed the VXLAN Network Identifier (VNI).
• This allows up to 16M VXLAN segments to coexist within the same
administrative domain.

4 Copyright © 2014 Juniper Networks, Inc.

 VXLAN Packet Format
• VXLAN uses a MAC in User Datagram Protocol (MAC-in-UDP)
encapsulation technique and a 24-bit segment identifier in the form of
a VXLAN ID.
F
OUTER OUTER OUTER VXLAN
Original L2 Frame C
MAC IP UDP Header S

48 DEST MAC
IP HDR SOURCE
72 16
DATA PORT FLAGS
48 SRC MAC 8
PROTO: VXLAN RRRR1RRR
8 16
VLAN UDP PORT
32
(OPTIONAL) UDP RESERVED 24
16 CKSUM 16
ETH TYPE LENGTH
16
0X0800 SRC IP: CHKSUM VNI 24
32 16
MY VTEP 0X0000
DST IP: RESERVED 8
32
DEST VTEP

5 Copyright © 2014 Juniper Networks, Inc.

 Control Plane for VXLAN
• Controller-less
1. Multicast Flood and Learn
2. EVPN-VXLAN (MP-BGP EVPN Signaling)

• Controller Based
1. Contrail with EVPN
2. Contrail with OVSDB
3. VMWare NSX

6 Copyright © 2014 Juniper Networks, Inc.

 EVPN-VXLAN
• Multiprotocol Border Gateway Protocol Ethernet Virtual Private Network (MP-BGP
EVPN) as the control plane for VXLAN.

• It provides VTEP peer discovery and end-host reachability information distribution.

• It allows more scalable VXLAN overlay network designs suitable for private and
public clouds.

• The MP-BGP EVPN control plane introduces a set of features that reduces or
eliminates traffic flooding in the overlay network and enables optimal forwarding
for both west-east and south-north traffic.

7 Copyright © 2014 Juniper Networks, Inc.

 L2 Gateway – VTEP capable of switching only IntraVNI

Spine-1 Spine-2
Leaf-1>show vlans bd1

Routing instance VLAN name Tag Interfaces

default-switch bd1 1 VM1 to VM4 goes over
et-0/0/67.0 the VTEP tunnel
vtep.32769*
vtep.32770*
root@cloud-bms-elite-sw01> show ethernet-switching table
vlan-id 1
Vlan MAC MAC Logical Active

name address flags interface

source VNID : 100 VNID : 200 VNID : 100
bd1 88:a2:5e:cc:d9:80 D vtep.32769 Leaf-1 Leaf-2 Leaf-3
100.0.0.12

VM1 to VM2 goes

through the switch:
No VXLAN involved VM1 VM2 VM3 VM4
VLAN :10 VLAN : 10 VLAN : 20 VLAN : 10

8 Copyright © 2014 Juniper Networks, Inc.

 L2 Gateway – Packet DMAC = S1MAC Vlan MAC Interface
Flow SMAC = L1MAC bd1 M1 vtep.32769
Vlan MAC Interface
SIP = L1IP
bd1 M1 vtep.32769
DIP = L3IP
UDP
VXLAN (VNI=100)
Inner L2 Spine-1 Spine-2
Vlan MAC Interface
bd1 M1 xe-0/0/0
VM1 to VM4 goes over
the VTEP tunnel

FF:FF:FF
:FF:FF:F
F
VNID : 100 VNID : 200 VNID Inner
: 100L2
M1 Leaf-1 Leaf-2 Leaf-3

0x0806

….. VM1 VM2 VM3 VM4

VLAN :10 VLAN : 10 VLAN : 20 VLAN :10
M1, IP1 M4, IP4

9 Copyright © 2014 Juniper Networks, Inc.

 L3 Gateway – InterVNI routing
IRB Config on Spine 1 IRB Config on Spine-2
irb {
unit 1 { irb {
family inet { Spine-2 unit 1 {
Spine-1 family inet {
address 1.0.1.251/24{
virtual-gateway-address 1.0.1.253; address 1.0.1.252/24{
} virtual-gateway-address 1.0.1.253;
} VRF }
VRF
unit 2 { }
family inet { unit 2 {
address 2.0.1.251/24{ family inet {
virtual-gateway-address 2.0.1.253; address 2.0.1.252/24{
} VNID : 100 virtual-gateway-address 2.0.1.253;
VNID : 200
} }
Leaf-1 Leaf-2 }
} VM1 to VM3 : Traffic dest
mac is IRB VGW Reaches } Red Lines are VTEP tunnels
}
over VTEP. } Blue lines are IntraVNI
VRF IP lookup puts the VTEP
traffic to VTEP which goes VM1 VM2 VM3
to leaf2 VLAN :10 VLAN : 20
VLAN :10
M2, IP2 M3, IP3
M1, IP1

10 Copyright © 2014 Juniper Networks, Inc.

 L3 Gateway- Virtual GW MAC Resolution Packet Flow
Spine-1 Spine-2
DMAC = L1MAC
SMAC = S1MAC

P
E
T
V
bd1 M1 vtep.zzz
bd1 M1 vtep.yyyy
SIP = S1IP VRF-1
VRF-1
DIP = L1IP IRB.1– 1.0.1.253, VMAC1
IRB.1– 1.0.1.253, VMAC1
IRB.2– 2.0.1.253, VMAC2 VTEP.
UDP IRB.2– 2.0.1.253, VMAC2
V
VXLAN (VNI=100) T
E Vlan MAC Interface
Inner L2-ARP Resp
Vlan MAC Interface P V bd1 VMAC1 esi.xxxx
V T V
bd2 VMAC2 esi.xxxx
bd1 VMAC1 esi.xxxx .
ET
T
DMAC = S1MAC E bd1 M1 vtep.xxx
bd2 VMAC2 esi.xxxx 3 P E P
bd1 M1 xe-0/0/0 SMAC = L1MAC2 P
7
SIP = L1IP 6
DIP = S1IP 9
FF:FF:FF:F
FF:FF:FF:F UDP
F:FF:FF
F:FF:FF
VXLAN (VNI=100)VNID : 100 VNID : 200
M1
M1 InnerInner Leaf-1
L2 Resp
L2-ARP Leaf-2

0x0806
0x0806

TIP:1.0.1.
TIP:1.0.1.
253
253 VM1 VM2 VM3
….. VLAN : 10 VLAN : 10
….. VLAN : 20
M1, IP1 M2, IP2 M3, IP3
1.0.1.1 1.0.1.2 2.0.1.1

11 Copyright © 2014 Juniper Networks, Inc.

 EVPN-VXLAN Topology/Config Design

12 Copyright © 2014 Juniper Networks, Inc.

 Topology
QFX5100-24Q QFX5100-24Q QFX5100-24Q QFX5100-24Q
Lo0:100.0.0.1 Lo0:100.0.0.2 Lo0:100.0.0.3 Lo0:100.0.0.4

12 15 14 13 12 15 14 13 14
14 13 13 12 12 15
15

58 59 61 60 58 59 61 60 61 60 59 58 61 60 58 59

QFX10002 QFX10002
QFX10002 QFX10002
Lo0:100.0.0.11 Lo0:100.0.0.12
Lo0:100.0.0.13 Lo0:100.0.0.14
IRB-vlan100: 10.1.100.1 66 66 67
67 66 67 67 66
IRB-vlan101: 10.1.101.1
IRB-vlan108: 10.1.108.1 50 51 50 51
50 51 51
50
QFX5100 52 QFX5100
QFX5100 Lo0:100.0.0.23 53
QFX5100 Lo0:100.0.0.24
Lo0:100.0.0.21 Lo0:100.0.0.22 52 53
12 13 QFX5100
12
L2 Switch
12 13 12 13 48
STC-8/5 STC-4/6 STC-8/6
STC-8/3 STC-1/3 STC-8/4 STC-1/4 STC-4/8

vlan100: 10.1.100.10-14 MAC: DEADBE:E100:10-14 vlan100: 10.1.100.40-44 MAC: DEADBE:E100:40-44

vlan101: 10.1.101.10-14 MAC: DEADBE:E101.10-14 vlan101: 10.1.101.40-44 MAC: DEADBE:E101.40-44
vlan100: 10.1.100.50-54 MAC: FACE:B0:0100:10-14
vlan101: 10.1.101.50-54 MAC: FACE:B0:0101:10-14 vlan100: 10.1.100.80-84 MAC: FACE:B0:0100:40-44
13 vlan108: 10.1.108.50-54 MAC: FACE:B0:0108:10-14 vlan101: 10.1.101.80-84 MAC: FACE:B0:0101:40-44
Copyright © 2014 Juniper Networks, Inc.
 set policy-options policy-statement bgp-ipclos-in term loopbacks from route-filter 100.0.0.0/16 orlonger

EBGP Underlay set policy-options policy-statement bgp-ipclos-in term loopbacks then accept
set policy-options policy-statement bgp-ipclos-out term loopback from protocol direct
set policy-options policy-statement bgp-ipclos-out term loopback from route-filter 100.0.0.11/32 orlonger
Fabric-1 Fabric-2 Fabric-3 Fabric-4
set policy-options policy-statement bgp-ipclos-out term loopback then community add MYCOMMUNITY
AS-60001 AS-60002 AS-60003 AS-60004
set policy-options policy-statement bgp-ipclos-out term loopback then next-hop self
Lo0:100.0.0.1 set policy-options Lo0:100.0.0.2 Lo0:100.0.0.3 Lo0:100.0.0.4
policy-statement bgp-ipclos-out
12
term loopback
15
then accept
12 15 14 12 13 14 13 14
15 14 13 set policy-options 13
policy-statement bgp-ipclos-out term 12
as-path from as-path asPathLength2 15
set policy-options policy-statement bgp-ipclos-out term as-path from community MYCOMMUNITY
set policy-options policy-statement bgp-ipclos-out term as-path then reject
set policy-options community MYCOMMUNITY members target:12345:111
58 59 61 59 as-path
60 set policy-options 61 asPathLength2 61 60 58 61 60 58
58 60 ".{2,}" 59 59
Spine-1 Spine-2 Spine-3 Spine-4
AS-60011 AS-60012 AS-60013 AS-60014
Spine Layer Lo0:100.0.0.11 Lo0:100.0.0.12 Lo0:100.0.0.13 Lo0:100.0.0.14
66 66 67
67 66 67 67 66

50 51 50 51
50 51 51
50
LEAF-1 LEAF-2 LEAF-3 LEAF-4
Leaf Layer AS-60021 AS-60022 AS-60023 AS-60024
Lo0:100.0.0.24
Lo0:100.0.0.21 Lo0:100.0.0.22 Lo0:100.0.0.23
set policy-options policy-statement bgp-ipclos-in term loopbacks from route-filter 100.0.0.0/16 orlonger
set policy-options policy-statement bgp-ipclos-in term loopbacks then accept

POD1 POD2
set policy-options policy-statement bgp-ipclos-out term loopback from protocol direct
set policy-options policy-statement bgp-ipclos-out term loopback from route-filter 100.0.0.21/32 exact
set policy-options policy-statement bgp-ipclos-out term loopback then next-hop self
set policy-options policy-statement bgp-ipclos-out term loopback then accept
set policy-options policy-statement bgp-ipclos-out term reject then reject

14 Copyright © 2014 Juniper Networks, Inc.

 IBGP Overlay AS-65200 set policy-options policy-statement OVERLAY-IN term reject-remote-gw from family evpn
set policy-options policy-statement OVERLAY-IN term reject-remote-gw from next-hop 100.0.0.13
set policy-options policy-statement OVERLAY-IN term reject-remote-gw from next-hop 100.0.0.14
set protocols bgp group overlay-evpn type internal set policy-options policy-statement OVERLAY-IN term reject-remote-gw from nlri-route-type 1
set protocols bgp group overlay-evpn local-address 100.0.0.14 set policy-options policy-statement OVERLAY-IN term reject-remote-gw from nlri-route-type 2
set policy-options policy-statement OVERLAY-IN term reject-remote-gw then reject
set protocols bgp group overlay-evpn family evpn signaling
set policy-options policy-statement OVERLAY-IN term accept-all then accept
set protocols bgp group overlay-evpn local-as 65200
set protocols bgp group overlay-evpn multipath
set protocols bgp group overlay-evpn neighbor 100.0.0.11
set protocols bgp group overlay-evpn neighbor 100.0.0.12
set protocols bgp group overlay-evpn neighbor 100.0.0.13 Full Mesh Full Mesh Full Mesh Full Mesh
IBGP IBGP IBGP IBGP
Spine-1 Spine-2 Spine-3 Spine-4
Lo0:100.0.0.11
RR Lo0:100.0.0.12
RR Lo0:100.0.0.13
RR Lo0:100.0.0.14
RR
Cluster:2.2.2. Cluster:2.2.2. Cluster:3.3.3. Cluster:3.3.3.
2 2 3 3
RR RR
RR RR
Client Client
Client Client LEAF-3 LEAF-4
LEAF-1 LEAF-2 Lo0:100.0.0.23 Lo0:100.0.0.24
Lo0:100.0.0.21 Lo0:100.0.0.22
set protocols bgp group overlay-evpn-rr type internal
set protocols bgp group overlay-evpn-rr local-address 100.0.0.14
set protocols bgp group overlay-evpn-rr family evpn signaling
set protocols bgp group overlay-evpn-rr cluster 3.3.3.3 POD1 POD2
set protocols bgp group overlay-evpn-rr local-as 65200
set protocols bgp group overlay-evpn-rr multipath
set protocols bgp group overlay-evpn-rr neighbor 100.0.0.23
set protocols bgp group overlay-evpn-rr neighbor 100.0.0.24
15 Copyright © 2014 Juniper Networks, Inc.
 Intra VLAN Traffic Type2 Tunnel
Fabric-1 Fabric-2 Fabric-3 Fabric-4

12 15 14 13 12 15 14 13 14
14 13 13 12 12 15
15

58 59 61 60 58 59 61 60 61 60 59 58 61 60 58 59

Spine-1 Spine-2 Spine-3 Spine-4

AS-60011 AS-60012 AS-60013 AS-60014

66 66 67
67 66 67 67 66
1.3
1.350 51
1.2 50 51
50 51 51
50
LEAF-3 52 LEAF-4
LEAF-1 AS-60023 53
LEAF-2 AS-60024
AS-60021 AS-60022 52 53
13 TOR
1.1 STC-8/5 STC-4/6 48
STC-8/3 STC-8/4 STC-1/4 STC-8/6
STC-1/3
POD1 POD2 STC-4/8

1. Traffic Flow
1.1 Hosts on same leaf
1.2 Hosts on different leaf but same POD
16 1.3 Hosts on different POD Copyright © 2014 Juniper Networks, Inc.
 Inter VLAN Traffic - Type2 Tunnel
Fabric-1 Fabric-2 Fabric-3 Fabric-4

12 15 14 13 12 15 14 13 14
14 13 13 12 12 15
15

58 59 61 60 58 59 61 60 61 60 59 58 61 60 58 59

Spine-1 Spine-2 Spine-3 Spine-4

AS-60011 AS-60012 AS-60013 AS-60014
VRF Routing
Instance 66 66 67
67 66 67 67 66

1.350 51 50 51
50 51 51
50
LEAF-3 52 LEAF-4
LEAF-1 AS-60023 53
1.1 AS-60021 1.2 LEAF-2 AS-60024
AS-60022 52 53
13 TOR
STC-8/5 STC-4/6 48
STC-8/3 STC-8/4 STC-1/4 STC-8/6
STC-1/3
POD1 POD2 STC-4/8

1. Traffic Flow
1.1 Inter VLAN on same leaf
1.2 Inter VLAN on different leaf but same POD
17 1.3 Inter VLAN on different POD Copyright © 2014 Juniper Networks, Inc.
 ESI set interfaces et-0/0/53 ether-options 802.3ad ae0
set interfaces ae0 esi 00:01:01:01:01:01:01:01:01:01
set interfaces ae0 esi all-active
Fabric-1 Fabric-2 Fabric-3 lacp active
set interfaces ae0 aggregated-ether-options Fabric-4
set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:01:01:01
12 15 14 13 12 15 14 13 14
set interfaces et-0/0/52 12 ae0 unit
set13interfaces 0 family ethernet-switching interface-mode
12 trunk 15
15 ether-options
14 13 802.3ad ae0
set interfaces ae0 esi 00:01:01:01:01:01:01:01:01:01 set interfaces ae0 unit 0 family ethernet-switching vlan members 100-108
set interfaces ae0 esi all-active
set interfaces ae0 aggregated-ether-options lacp active
58 59 61 60 lacp system-id
58 59 61 60 61 60 59 58 61 60 58 59
set interfaces ae0 aggregated-ether-options 00:00:00:01:01:01
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
Spine-1 Spine-2 Spine-3 Spine-4
set interfaces ae0 unit 0 family ethernet-switching vlan members 100-108
AS-60011 AS-60012 AS-60013 AS-60014

66 66 67
67 66 67 67 66

50 51 50 51
50 51 51
50
LEAF-3 52 LEAF-4
set interfaces xe-0/0/48:0 unit 0 family ethernet-switching interface-mode trunk 53
LEAF-1 LEAF-2 vlan members 100-108 AS-60023 AS-60024
set interfaces xe-0/0/48:0
AS-60021
unit 0 family ethernet-switching 52 AE 53
set interfaces et-0/0/52 ether-options 802.3ad ae0 AS-60022
set interfaces et-0/0/53 ether-options 802.3ad ae0 13 TOR
set interfaces ae0 aggregated-ether-options lacp active STC-8/5 STC-4/6 48
set interfaces ae0 unit 0 family
STC-8/3 ethernet-switching interface-mode
STC-8/4 trunk
STC-1/4 STC-8/6
STC-1/3
set interfaces ae0 unit 0 family ethernet-switching vlan members 100-108
POD1 POD2 STC-4/8

root@cloud-bms-elite-sw03> show evpn instance esi 00:01:01:01:01:01:01:01:01:01

designated-forwarder
Instance: default-switch
Number of ethernet segments: 8164
ESI:00:01:01:01:01:01:01:01:01:01
Designated forwarder: 100.100.255.4 Copyright © 2014 Juniper Networks, Inc.
18
 L2GW and L3GW Collapsed into one device

19 Copyright © 2014 Juniper Networks, Inc.

 L2 and L3 GW Collapsed
Spine-1 Spine-2

Spine can be just RR.

Spine will not have
any VXLAN config

• VRF and IRB configured in Leaf IRB IRB

itself. VG: 100
VNID VG
• Both L2 and L3 GW VNID : 200
functionality is taken care in
Leaf itself.
• VTEP tunnels are between
Leaf only
VM1 VM2 VM3
VLAN : 10 VLAN : 10 VLAN : 20

20 Copyright © 2014 Juniper Networks, Inc.

 MAC + IP Route
Spine-1 Spine-2
1.VM1 MAC gets
advertised from Leaf1 to
Leaf2
1. Leaf1: ARP Resolved 2. If there is a need then
for VM1 Leaf2 have to resolve the
2. ARP Response to VM1 Leaf-1 Leaf-2 ARP
With MAC+IP support:

IRB IRB 1.VM1 MAC and MAC+IP

gets advertised from
VG VG Leaf1 to Leaf2
VNID : 100 VNID : 200 2 Leaf2: ARP Resolved for
VM1

VM1: ARP req VM1 VM2 VM3

Leaf1 VG
VLAN : 10 VLAN : 10 VLAN : 20

21 Copyright © 2014 Juniper Networks, Inc.

 L3 Gateway Configuration Models
Option - 1 Option - 2
irb { irb {
unit 1 { unit 1 {
family inet { family inet {
address 1.0.1.251/16 { address 1.0.1.251/16;
virtual-gateway-address 1.0.1.253; }
} mac 00:00:ba:ba:00:01
} }

22 Copyright © 2014 Juniper Networks, Inc.

 VMTO - VM Mobility Traffic Optimizer

This is taken care by configuring the

same VGW on leaf

VGW:10.10.1 VGW:
VGW:
10.10.10.2
0.1
10.10.10.1

If Server 2 moves from DC2 to DC3,

Server2 VGA is still pointing to DC2 VGW.
So Intervlan Traffic reaches Leaf on DC2

23 Copyright © 2014 Juniper Networks, Inc.

 VMTO - VM L3VPN-PE1
root@cloud-fabric-elit-01# run show route table VRF1 match-prefix 10.1.*.40
10.1.46.40/32 *[BGP/170] 00:01:48, localpref 100, from 100.100.255.4
Mobility Traffic STC-1/1 L3VPN
QFX10002
cloud-fabric-elit-01
AS path: I, validation-state: unverified
> to 172.16.0.57 via et-0/0/32.0, label-switched-path rr1-to-u8
Optimizer VRF
4
Lo0:100.0.0.31
14 22 32
10.1.47.40/32 *[BGP/170] 00:01:48, localpref 100, from 100.0.0.13
AS path: I, validation-state: unverified
> to 172.16.0.55 via et-0/0/22.0, label-switched-path rr1-to-elit3

iBGP with family inet-vpn and 10.1.46.0/24 *[BGP/170] 00:44:14, localpref 100, from 100.0.0.13
LDP/RSVP AS path: I, validation-state: unverified
> to 172.16.0.55 via et-0/0/22.0, label-switched-path rr1-to-elit3
[BGP/170] 5d 04:00:03, localpref 100, from 100.100.255.4
AS path: I, validation-state: unverified
> to 172.16.0.57 via et-0/0/32.0, label-switched-path rr1-to-u8

LEAF1 LEAF2 LEAF3 LEAF4

6 16 6 16 6 16 0/0/7
34:1 1/0/7 0/0/12
QFX10002 QFX10002
STC-1/8 QFX10002 QFX10008
cloud-bms-elite-sw01 cloud-bms-elite-sw02 12
cloud-bms-elite-sw03 cloud-bms-ultimat-sw01 STC-1/7
Lo0:100.0.0.11 Lo0:100.0.0.12 0/0/34:0
Lo0:100.0.0.13 Lo0:100.100.255.4
EVPN EVPN
67EVPN
66 0/0/26 EVPN VRF 1/0/26
VRF 67 66
VRF
67 34:0 34:0 66

STC-1/6
VRF
STC-1/5
51 50
51
root@cloud-bms-ultimat-sw01# run
50 show
51 arp no-resolve vpn VRF_10
50 50 51
QFX5100
MAC Address Address Interface Flags
QFX5100
root@cloud-bms-elite-sw03# run show arp no-resolve vpn VRF_10 de:ad:be:00:2e:40 10.1.46.40 irb.46 [ae203.0]
cloud-bms-sw04
QFX10008
none
QFX5100 cloud-bms-elite-sw04
Lo0:100.0.0.23 de:ad:be:00:2f:40 10.1.47.40 irb.47 [ae203.0]
cloud-bms-sw01 cloud-bms-sw05 Lo0:100.0.0.24permanent remote
MAC Address
Lo0:100.0.0.21 Address Interface
Lo0:100.0.0.22 Flags
12 13
de:ad:be:00:2e:40 10.1.46.40 irb.46 [ae203.0] permanent remote 12
12 de:ad:be:00:2f:40
13 12 10.1.47.40 13irb.47 [ae203.0] none
STC-8/5 STC-4/6 STC-4/8 STC-8/6
STC-8/3 STC-1/3 STC-8/4 STC-1/4

24 Copyright © 2014 Juniper Networks, Inc.

 EVPN withType-5
Routes
for Inter-DC / Inter-POD Connectivity

25 Copyright © 2014 Juniper Networks, Inc.

 Why EVPN Type-5 route?
What is the problem we are trying to solve ?
• Customers having multiple IP subnets (IRB) in a VRF instance (Tenant)
need to securely route between them across POD’s within DC or
between Datacenters
• EVPN/VXLAN only distributes Layer-2 MAC’s + IP across POD’s using
EVPN Type-2 EVPN routes within the Datacenter
• EVPN-Type-5 route will carry the IP prefixes across POD’s /DC’s
• IP prefix reachability could be achieved using traditional MPLS
L3VPN, but It requires MPLS core and additional NLRI ( inet-vpn) to
be enabled on the Spines.

26 Copyright © 2014 Juniper Networks, Inc.

 EVPN-VXLAN (Type-5)

27 Copyright © 2014 Juniper Networks, Inc.

 L3VPN VS EVPN/Type-5
MP-iBGP session “inet-vpn unicast” Irb.3 ->20.3.0.0/16
Irb.3 ->20.1.0.0/16
Irb.4-> 20.4.0.0/16
Irb.4-> 20.2.0.0/16
CPE-1 CPE-2
Site 2 PE-1 PE-2 Site 1
MPLS Core
VRF VRF
CPE-3 CPE-4
Site 1 VRF VRF Site 2

Irb.1 ->10.1.0.0/16 MPLS-LSP for all VRF’s Irb.1 ->10.3.0.0/16

Irb.2-> 10.2.0.0/16 Irb.2-> 10.4.0.0/16

Irb.3 ->20.1.0.0/16 MP-iBGP session “evpn-signaling”

Irb.4-> 20.2.0.0/16 Irb.3 ->20.3.0.0/16
CPE-1 Irb.4-> 20.4.0.0/16
CPE-2
Site 2 PE-1 IP-Core PE-2 Site 1
VRF VRF
CPE-3 CPE-4
Site 1 VRF VRF Site 2
Type-5 Tunnel for VRF RED

Irb.1 ->10.1.0.0/16 Irb.1 ->10.3.0.0/16

Irb.2-> 10.2.0.0/16 Type-5 Tunnel for VRF Blue Irb.2-> 10.4.0.0/16

Type-5 Tunnel = Spine X VRF’s

28 Copyright © 2014 Juniper Networks, Inc.

 EVPN Type-5 EVPN BGP Global Config
• set protocols bgp group Inter-POD-EVPN type external
• set protocols bgp group Inter-POD-EVPN multihop
• set protocols bgp group Inter-POD-EVPN local-address 100.100.255.2
• set protocols bgp group Inter-POD-EVPN family evpn signaling
• set protocols bgp group Inter-POD-EVPN neighbor 100.100.255.3 peer-as 65201
• set protocols bgp group Inter-POD-EVPN neighbor 100.100.255.4 peer-as 65201

29 Copyright © 2014 Juniper Networks, Inc.

 EVPN Type-5 VRF Config
• set routing-instances VRF_lite1 protocols evpn ip-prefix-routes advertise direct-nexthop
• set routing-instances VRF_lite1 protocols evpn ip-prefix-routes encapsulation vxlan
• set routing-instances VRF_lite1 protocols evpn ip-prefix-routes vni 1677
• set routing-instances VRF_lite1 protocols evpn ip-prefix-routes export EVPN-TYPE5-EXPORT-VRF_lite1

• show policy-options policy-statement EVPN-TYPE5-EXPORT-VRF_lite1 | display set

• set policy-options policy-statement EVPN-TYPE5-EXPORT-VRF_lite1 term 1 from route-filter 10.1.0.0/16

exact
• set policy-options policy-statement EVPN-TYPE5-EXPORT-VRF_lite1 term 1 then accept

30 Copyright © 2014 Juniper Networks, Inc.

 EVPN-VXLAN Type-5
and Type-2 on the same
topology

31 Copyright © 2014 Juniper Networks, Inc.

 L2 & L3 GW Collapsed with Type-5 ① External route advertised to Leaf3 and Leaf4
over BGP
External Router STC-8/5
Lo0:100.0.0.31

② External route converted as Type-5 and

advertised from Leaf3 and Leaf4 to RR-1 and
RR-2
RR-1 RR-2
Lo0:100.0.0.1 Lo0:100.0.0.2

③ Type-5 route advertised from RR-1 and RR-2 to

Leaf1 and Leaf2 Leaf1 Leaf2 Border-Leaf2 Border-Leaf1
QFX10002 QFX10002
STC-1/8 QFX10002 QFX10008 STC-1/7
Lo0:100.0.0.11 Lo0:100.0.0.12
Lo0:100.0.0.13 Lo0:100.0.0.14

root@cloud-bms-ultimat-sw01# show routing-instances VRF_1 | display set | display inheritance

QFX5100set routing-instances VRF_1 instance-type vrf
QFX5100 set routing-instances VRF_1 interface irb.1 QFX10002
QFX5100 Lo0:100.0.0.23
set routing-instances VRF_1 interface lo0.1001 Lo0:100.0.0.24
Lo0:100.0.0.21 Lo0:100.0.0.22 set routing-instances VRF_1 interface irb.4001
set routing-instances VRF_1 instance-type vrf set routing-instances VRF_1 route-distinguisher 100.100.255.4:1
set routing-instances VRF_1 interface irb.1 set routing-instances VRF_1 vrf-target target:1:1
set routing-instances VRF_1 interface lo0.1 STC-8/5 setSTC-4/6
routing-instances VRF_1 protocols bgp group mx type external STC-8/6
STC-4/8
STC-8/3
set routing-instances VRF_1 route-distinguisher 100.0.0.11:1
STC-1/3 STC-8/4 STC-1/4 set routing-instances VRF_1 protocols bgp group mx import set-pref
set routing-instances VRF_1 vrf-target target:1:1 set routing-instances VRF_1 protocols bgp group mx family inet unicast
set routing-instances VRF_1 protocols bgp group mx export direct
set routing-instances VRF_1 routing-options static route 100.0.0.15/32 discard
set routing-instances VRF_1 protocols bgp group mx peer-as 65000
set routing-instances VRF_1 protocols evpn ip-prefix-routes advertise direct-nexthop
set routing-instances VRF_1 protocols bgp group mx local-as 65200
set routing-instances VRF_1 protocols evpn ip-prefix-routes encapsulation vxlan set routing-instances VRF_1 protocols bgp group mx multipath
set routing-instances VRF_1 protocols evpn ip-prefix-routes vni 10001 set routing-instances VRF_1 protocols bgp group mx neighbor 10.1.211.2
set routing-instances VRF_1 protocols evpn ip-prefix-routes export DummyRoute set routing-instances VRF_1 protocols evpn ip-prefix-routes advertise direct-nexthop
set routing-instances VRF_1 protocols evpn ip-prefix-routes encapsulation vxlan
set routing-instances VRF_1 protocols evpn ip-prefix-routes vni 1677
32 Copyright © 2014 Juniper Networks, Inc.
 EVPN-VXLAN troubleshooting

33 Copyright © 2014 Juniper Networks, Inc.

 EVPN Type-5 Verification
• root@SPINE-2# run show evpn ip-prefix-database
• root@SPINE-2# run show route table VRF_lite1.evpn.0

• TFXPC0(vty)# sh shim jnh vtep-ip-table

• FXPC0(vty)# sh shim jnh tunnel-nh-table
• FXPC0(vty)# show shim jnh vxlan-data
•

34 Copyright © 2014 Juniper Networks, Inc.

 EVPN-VXLAN troubleshooting
# Serviceability and diagnose-ability

Existing L2 bridge commands are enhanced to provide VXLAN information's.

1. Display MAC address learnt from remote VTEP

- show ethernet-switching table
- show ethernet-switching vxlan-tunnel-end-point remote mac-table
2. Display VTEP association to VLAN
- show vlans
3. Display remote VTEP information
- show ethernet-switching vxlan-tunnel-end-point remote ip <remote vtep ip>
4. Display EVPN database information
- show evpn datbase
5. Display MAC and MAC+IP route
- show route tabel bgp.evpn.0
6. Display ARP Learning Local or Remote
- show arp no-resolve expiration-time

35 Copyright © 2014 Juniper Networks, Inc.

 EVPN-VXLAN troubleshooting
Ideal case
• Underlay routing protocols should be up and running on VTEP endpoints

• L2 instance should be UP and VXLAN is associated properly with the L2 Bridging instance to form a VXLAN
tunnels

• VTEP interfaces(both Source and Remote) should be UP

• L2 MAC learning in local L2 interface as well as over VTEP interface should happen

• IRB interface should be UP and ARP should be resolved for L3 traffic coming from Internet over IRB to
VTEP endpoints or vice versa

36 Copyright © 2014 Juniper Networks, Inc.

EVPN-VXLAN Topology RR-1 RR-2
for Trouble shooting
QFX10002 QFX10002
cloud-fabric-elit-01 cloud-msdc-bms-sw05
Lo0:100.0.0.31 Lo0:100.0.0.32
4 14 22 32 22 14 4 32

6 16 6 16 6 16 0/0/7
1/0/7
QFX10002 QFX10002
QFX10002 QFX10008
cloud-bms-elite-sw01 cloud-bms-elite-sw02 12
cloud-bms-elite-sw03 cloud-bms-ultimat-sw01
Lo0:100.0.0.11 Lo0:100.0.0.12
Lo0:100.0.0.13 Lo0:100.100.255.4
IRB-vlan1: 10.1.1.1 67 0/0/26
67 67 66
IRB-vlan2: 10.1.2.1

50 51
50
QFX5100 QFX5100 QFX5100
cloud-bms-sw05 cloud-bms-sw01 cloud-bms-sw04
Lo0:100.0.0.22 Lo0:100.0.0.22 Lo0:100.0.0.23

10G Links

vlan1: 10.1.1.15 MAC: 88:a2:5e:cc:48:80

vlan1: 10.1.1.25 MAC: 88:a2:5e:cb:a8:80

vlan1: 10.1.1.5 MAC: 88:a2:5e:cc:d9:80

37 Copyright © 2014 Juniper Networks, Inc.
 cloud-bms-elite-sw01 Logs

root@cloud-bms-elite-sw01> show arp no-resolve

MAC Address Address Interface Flags
88:a2:5e:cc:d9:80 10.1.1.5 irb.1 [vtep.32769] permanent remote --- REMOTE NON-ESI
88:a2:5e:cc:48:80 10.1.1.15 irb.1 [et-0/0/67.0] none ---- Local
88:a2:5e:cb:a8:80 10.1.1.25 irb.1 [.local..3] permanent remote -- REMOTE ESI

root@cloud-bms-elite-sw01> show ethernet-switching table vlan-id 1

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)
Ethernet switching table : 4 entries, 4 learned
Routing instance : default-switch
Vlan MAC MAC Logical Active
name address flags interface source
bd1 00:00:5e:00:02:01 DR,SD esi.92140 05:00:00:00:00:00:00:03:e9:00
bd1 88:a2:5e:cb:a8:80 DR esi.92303 00:04:04:04:04:04:04:04:04:04
bd1 88:a2:5e:cc:48:80 D et-0/0/67.0
bd1 88:a2:5e:cc:d9:80 D vtep.32769 100.0.0.12

38 Copyright © 2014 Juniper Networks, Inc.

 root@cloud-bms-elite-sw01:~ # cprod -A fpc0 -c "show shim bridge fdb table all" | grep 88:a2
88:a2:5e:cb:a8:80 262146 esi.92303 Remote DA/RE Instl 0x000000 0x 0x 0
000000 No 0x4000000000000000030000000000000000000000
88:a2:5e:cc:48:80 262146 et-0/0/67.0 Local RE/DA/SA Instl 0x000004 0x
0x71022d48 000001 No 0x00000001
88:a2:5e:cc:d9:80 262146 vtep.32769 Remote DA/RE Instl 0x000000 0x 0x 0
000000 No 0x4000000000000000000000000000000000000000

39 Copyright © 2014 Juniper Networks, Inc.

 root@cloud-bms-elite-sw01> show evpn database extensive esi 00:04:04:04:04:04:04:04:04:04
Instance: default-switch

VN Identifier: 1001, MAC address: 88:a2:5e:cb:a8:80

Source: 00:04:04:04:04:04:04:04:04:04, Rank: 1, Status: Active
Remote origin: 100.0.0.13
Remote origin: 100.100.255.4
Timestamp: Sep 13 14:45:02 (0x57d8735e)
State: <Remote-To-Local-Adv-Done>
IP address: 10.1.1.25
Remote origin: 100.100.255.4---------------------- Advertising PE

root@cloud-bms-elite-sw01> show evpn database extensive interface et-0/0/67.0

Instance: default-switch

VN Identifier: 1001, MAC address: 88:a2:5e:cc:48:80

Source: et-0/0/67.0, Rank: 1, Status: Active
Timestamp: Sep 13 14:31:12 (0x57d87020)
State: <Local-MAC-Only Local-To-Remote-Adv-Allowed>
IP address: 10.1.1.15
L3 route: 10.1.1.15/32, L3 context: VRF_1 (irb.1)

40 Copyright © 2014 Juniper Networks, Inc.

 root@cloud-bms-elite-sw01> show evpn database extensive
Instance: default-switch
VN Identifier: 1001, MAC address: 88:a2:5e:cb:a8:80
Source: 00:04:04:04:04:04:04:04:04:04, Rank: 1, Status: Active
Remote origin: 100.0.0.13
Remote origin: 100.100.255.4
Timestamp: Sep 13 14:45:02 (0x57d8735e)
State: <Remote-To-Local-Adv-Done>
IP address: 10.1.1.25
Remote origin: 100.100.255.4

VN Identifier: 1001, MAC address: 88:a2:5e:cc:48:80

Source: et-0/0/67.0, Rank: 1, Status: Active
Timestamp: Sep 13 14:31:12 (0x57d87020)
State: <Local-MAC-Only Local-To-Remote-Adv-Allowed>
IP address: 10.1.1.15
L3 route: 10.1.1.15/32, L3 context: VRF_1 (irb.1)

VN Identifier: 1001, MAC address: 88:a2:5e:cc:d9:80

Source: 100.0.0.12, Rank: 1, Status: Active
Timestamp: Sep 13 14:34:25 (0x57d870e1)
State: <Remote-To-Local-Adv-Done>
IP address: 10.1.1.5

41 Copyright © 2014 Juniper Networks, Inc.

 root@cloud-bms-elite-sw01> show route forwarding-table destination 10.1.1.25 table VRF_1
Routing table: VRF_1.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
10.1.1.25/32 dest 0 45:0:0:32:0:0 ucst 104711 1

root@cloud-bms-elite-sw01> show route forwarding-table destination 10.1.1.5 table VRF_1

Routing table: VRF_1.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
10.1.1.5/32 dest 0 88:a2:5e:cc:d9:80 ucst 104709 1 vtep.32769

root@cloud-bms-elite-sw01> show route forwarding-table destination 10.1.1.15 table VRF_1

Routing table: VRF_1.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
10.1.1.15/32 dest 0 88:a2:5e:cc:48:80 ucst 104710 1 et-0/0/67.0

42 Copyright © 2014 Juniper Networks, Inc.

 root@cloud-bms-elite-sw01> show route forwarding-table destination 10.1.1.25 table VRF_1
Routing table: VRF_1.inet
Internet:
Destination Type RtRef Next hop Type Index NhRef Netif
10.1.1.25/32 dest 0 45:0:0:32:0:0 ucst 104711 1

43 Copyright © 2014 Juniper Networks, Inc.

 BGP OUTPUT

root@cloud-bms-elite-sw01> show route table bgp.evpn.0 evpn-mac-address 88:a2:5e:cb:a8:80

bgp.evpn.0: 99128 destinations, 99128 routes (99128 active, 0 holddown, 0 hidden)

+ = Active Route, - = Last Active, * = Both

2:100.100.255.4:5000::1001::88:a2:5e:cb:a8:80/304
*[BGP/170] 00:40:28, localpref 100, from 100.0.0.31
AS path: I, validation-state: unverified
> to 172.16.0.50 via et-0/0/6.0
2:100.100.255.4:5000::1001::88:a2:5e:cb:a8:80::10.1.1.25/304
*[BGP/170] 00:40:28, localpref 100, from 100.0.0.31
AS path: I, validation-state: unverified
> to 172.16.0.50 via et-0/0/6.0

44 Copyright © 2014 Juniper Networks, Inc.

 PFE OUTPUT
TFXPC0(vty)# show route ip table VRF_1 prefix 10.1.1.15
IPv4 Route Table 5, VRF_1.5, 0x41000:
Destination NH IP Addr Type NH ID Interface
--------------------------------- --------------- -------- ----- ---------
10.1.1.15 irb_arp_ndp 104710 RT-ifl 582 irb.1 ifl 582

TFXPC0(vty)# show shim jnh arp-ndp-nh-table

NH ID IRB ARP_NDP ENTRY CHILD-NH Packets
------ ------------------ -------- -------
104709 0x77f48570 87740 0 0 0 0 0 0
104710 0x77f48860 75332 0 0 0 0 0 0
104711 0x7018bea8 92303 0 0 0 0 0 0

TFXPC0(vty)# show route ip table VRF_1 prefix 10.1.1.5

IPv4 Route Table 5, VRF_1.5, 0x41000:
Destination NH IP Addr Type NH ID Interface
--------------------------------- --------------- -------- ----- ---------
10.1.1.5 irb_arp_ndp 104709 RT-ifl 582

TFXPC0(vty)# show route ip table VRF_1 prefix 10.1.1.25

IPv4 Route Table 5, VRF_1.5, 0x41000:
Destination NH IP Addr Type NH ID Interface
--------------------------------- --------------- -------- ----- ---------
10.1.1.25 irb_arp_ndp 104711 RT-ifl 582
45 Copyright © 2014 Juniper Networks, Inc.
TFXPC0(vty)# show shim jnh arp-ndp-nh-table 104711
refcount :1
NH :0x73dd3340
NH Id :104711
n_nhs :1
Child NH Id :92303
Egress NH Id:0
arp_nd_vxlan:1
IRB IFL Idx :582
L2 IFL Idx :546
e_dmac :88:a2:5e:cb:a8:80
e_smac :0:31:46:7b:e1:18
e_n_vlantags :1
e_vlan_tag :1, 0
e_inner_tpid :0x0
e_outer_tpid :0x8100
num_tokens :1
tokens :2147547972, 0
jnh handle : 0x6e9ee908
======== PFE instance 0 ========
-------- ARP NDP handle:
handle 0x6e9ee908, flags 0x0, refcount 1
NH installed at addr NH 69914, INT_SEQ 0x100888d1
Raw dump of the nh words of size 1 words
0x110570e3
SEQ: [110570e3] Interm SIZE: 1 NO_ACT 1, USE_REMAP 0, NEXT_ADDR: 0ae1c, SZ: 3
Nexthop points to:
46 Copyright © 2014 Juniper Networks, Inc.
SEQ: [1004c619] Interm SIZE: 3 NO_ACT 0, USE_REMAP 0, NEXT_ADDR: 098c3, SZ: 1
 TFXPC0(vty)# show shim jnh descriptor dev 0 nh-mm-index 0x0ae1c 3
SEQ: [1004c619] Interm SIZE: 3 NO_ACT 0, USE_REMAP 0, NEXT_ADDR: 098c3, SZ: 1

ACT: [040301c8] EgNHId: EG_NHID: 0301c8

ACT: [603e9203] StBytes: OW byte 3 data 0xe9, OW byte 2 data 0x3

TFXPC0(vty)# show shim jnh descriptor dev 0 nh-mm-index 0x098c3 1

SEQ: [11073ac2] Interm SIZE: 1 NO_ACT 1, USE_REMAP 0, NEXT_ADDR: 0e758, SZ: 2

TFXPC0(vty)# show shim jnh descriptor dev 0 nh-mm-index 0xe758 2

SEQ: [400c3822] Eq_List SIZE: 2 FIN 0, BASE ADDR/SZ 0x18704/2, HASH 0 MASK: SZ 3, SEL 0, PTR/MASK
0xc000 OFST 0

TFXPC0(vty)# show shim jnh descriptor dev 0 nh-mm-index 0x18704 2

SEQ: [1004ff21] Interm SIZE: 2 NO_ACT 0, USE_REMAP 0, NEXT_ADDR: 09fe4, SZ: 1

ACT: [64010000] SrcID: srcid 0 upd 1

TFXPC0(vty)# show shim jnh descriptor dev 0 nh-mm-index 0x18706 2

SEQ: [1001f8d9] Interm SIZE: 2 NO_ACT 0, USE_REMAP 0, NEXT_ADDR: 03f1b, SZ: 1
ACT: [64010001] SrcID: srcid 1 upd 1

47 Copyright © 2014 Juniper Networks, Inc.

 TFXPC0(vty)# show nhdb id 116326 recursive
104711(Compst, IPv4, ifl:0:-, pfe-id:0, comp-fn:Chain)
529251 (Indirect, VPLS, ifl:0:-, pfe-id:0, i-ifl:0:-)
92303 (Compst, BRIDGE, ifl:0:-, pfe-id:0, comp-fn:Load Balance)
91477 (Compst, BRIDGE, ifl:0:-, pfe-id:0, comp-fn:Vxlan Unicast Encapsulation NH)
528425 (Indirect, IPv4, ifl:6159:et-0/0/6.0, pfe-id:3, i-ifl:0:-)
525753(Unilist, IPv4, ifl:0:-, pfe-id:3)
75334(Unicast, IPv4, ifl:6159:et-0/0/6.0, pfe-id:3)
99289(Unicast, IPv4, ifl:6160:et-0/0/16.0, pfe-id:4)
100707 (Compst, BRIDGE, ifl:0:-, pfe-id:0, comp-fn:Vxlan Unicast Encapsulation NH)
529655 (Indirect, IPv4, ifl:6159:et-0/0/6.0, pfe-id:3, i-ifl:0:-)
525753(Unilist, IPv4, ifl:0:-, pfe-id:3)
75334(Unicast, IPv4, ifl:6159:et-0/0/6.0, pfe-id:3)
99289(Unicast, IPv4, ifl:6160:et-0/0/16.0, pfe-id:4)

48 Copyright © 2014 Juniper Networks, Inc.

 ID Type Interface Next Hop Addr Protocol Encap MTU Flags PFE internal Flags

….

Egress Next-hop on pfe fe_id 0, port_num 0:

------------------------------------

L2 descriptor

==============

des-type p_next Shared? Primary Last desc_count app-type des-addr

des-size des-size

-------- ----- ------- -------- -------- -------- -------- --------

Private No Yes(1 )0 4 1 0 0x3c14c

Des addrs : 0x3c14c

F: P_CONST bv_flags: 2 address: 6

[ 6] Refcount 7207 SMAC 00:31:46:7b:e1:18

T: DMAC dmac: 000088a25ecba880

F: P_NEXT p_next: 277817

F: IDX_P_NEXT base_addr: 304748 entry_sz: 2 field_sel: 1 msb: 15 lsb: 0

F: COUNTER counter: 82007 cix_tc_en: 0

Flabel descr for App "DEFAULT" Flabel_id: 197064:, port_num 0

============================================================

des-type p_next Shared? Primary Last desc_count app-type des-addr

des-size des-size

-------- ----- ------- -------- -------- -------- -------- --------

Public No No (0 )3 2 1 0 0xb9e8

Des addrs : 0xb9e8

Flabel : 197064 Segment table index: : 48[4] Page Table index : 140[567] desc start addr:: 47592[1]

F: COUNTER counter: 94387 cix_tc_en: 0

F: P_NEXT p_next: 246092

ctr_idx 0x170b3, 0 pkts, 0 bytes

49 Copyright © 2014 Juniper Networks, Inc.

 cloud-bms-elite-sw03 Logs

root@cloud-bms-elite-sw03> show arp no-resolve

MAC Address Address Interface Flags
88:a2:5e:cc:d9:80 10.1.1.5 irb.1 [vtep.32770] permanent remote
88:a2:5e:cc:48:80 10.1.1.15 irb.1 [vtep.32769] permanent remote
88:a2:5e:cb:a8:80 10.1.1.25 irb.1 [ae202.0] permanent remote
ac:4b:c8:42:85:cc 10.1.201.2 irb.4001 [et-0/0/57.0] none
ac:4b:c8:42:85:cc 10.1.202.2 irb.4002 [et-0/0/57.0] none
5c:5e:ab:79:42:81 10.94.191.254 em0.0 none
fe:00:00:00:00:80 128.0.0.16 bme0.0 permanent
88:a2:5e:cd:4c:bf 172.16.0.20 et-0/0/59.0 none
20:4e:71:04:3c:85 172.16.0.54 et-0/0/42.0 none
•

50 Copyright © 2014 Juniper Networks, Inc.

 cloud-bms-ultimat-sw01 Logs

root@cloud-bms-ultimat-sw01-1> show arp no-resolve

MAC Address Address Interface Flags
88:a2:5e:cc:d9:80 10.1.1.5 irb.1 [vtep.32770] permanent remote
88:a2:5e:cc:48:80 10.1.1.15 irb.1 [vtep.32769] permanent remote
88:a2:5e:cb:a8:80 10.1.1.25 irb.1 [ae202.0] none
ac:4b:c8:42:87:16 10.1.211.2 irb.4001 [et-1/0/0.0] none
ac:4b:c8:42:87:16 10.1.212.2 irb.4002 [et-1/0/0.0] none

root@cloud-bms-ultimat-sw01-1> show ethernet-switching table vlan-id 1

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)

Ethernet switching table : 3 entries, 3 learned

Routing instance : default-switch
Vlan MAC MAC Logical Active
name address flags interface source
bd1 88:a2:5e:cb:a8:80 DL ae202.0
bd1 88:a2:5e:cc:48:80 D vtep.32769 100.0.0.11
bd1 88:a2:5e:cc:d9:80 D vtep.32770 100.0.0.12

51 Copyright © 2014 Juniper Networks, Inc.

 Collecting the Logs

L2ald
set protocols l2-learning traceoptions file l2ald.log
set protocols l2-learning traceoptions file size 1g
set protocols l2-learning traceoptions level all
set protocols l2-learning traceoptions flag all

RPD
set protocols evpn traceoptions file evpn.log
set protocols evpn traceoptions file size 1g
set protocols evpn traceoptions flag all

Kernel
rtsockmon –rt

52 Copyright © 2014 Juniper Networks, Inc.

 PFE Logs
show ukern_trace handles
38 EXPR_L2 terse Off On 65536 0
40 EXPR_IF terse Off On 65536 34
54 NH-HALP terse Off On 1048576 0
56 EXPR_VIRTUAL terse Off On 1048576 0
48 EXPR_PKT_DBG terse Off On 1048576 2
49 EXPR_PKT_ERR terse Off On 1048576 0
50 EXPR_PKT_TRACE terse Off On 1048576 0
51 EXPR_PKT_BUF terse Off On 10485
debug halp-pkt rx start (to enable host path pkt rx trace)
show ukern_trace 56
show ukern_trace 54
show ukern_trace 38
show ukern_trace 40
show ukern_trace 48
show ukern_trace 49
show ukern_trace 50
show ukern_trace 51

53 Copyright © 2014 Juniper Networks, Inc.

 Supportability
Platform L2GW L3GW L2/L3 L2/L3 EVPN Type-
Collapsed Collapsed 5
and EVPN
Type-5

QFX5100 Yes NS NS NS NS
QFX10K Yes Yes Yes Yes Yes

NS – Not Supported

54 Copyright © 2014 Juniper Networks, Inc.

 Reference Docs:
• Solution Document:
• PDT Test report :
https://systest.juniper.net/twiki/pub/Main/15_1_XD60/15.1X53.D60EV
PN-VXLAN-Summary-V1.
docx

55 Copyright © 2014 Juniper Networks, Inc.

Thank You
Backup Slides
 VXLAN Packet Format

F
OUTER OUTER OUTER VXLAN
Original L2 Frame C
MAC IP UDP Header S

IP Header UDP Header VXLAN Header

Ethernet Header
Protocol – Set 0×11 to indicate that the Source Port – Hash of the fields from the VXLAN Flags – Reserved bits set to
Destination Address – This is set to the frame contains a UDP packet inner packet. Set by transmitting VTEP. zero except bit 3, the I bit, which is set
MAC address of the destination VTEP if it to 1 to for a valid VNI. Packet dropped
is local of to that of the next hop device, Source IP – IP address of originating VTEP VXLAN Port – IANA assigned VXLAN Port if the flags are not recogonized.
usually a router, when the destination 4789.
VTEP is on a different L3 network. Destination IP – IP address of target VTEP VNI – 24-bit field that is the VXLAN
for unicast packets. Multicast group IP UDP Checksum – This should be set to Network Identifier
VLAN – This is optional address for broadcast, unknown unicast 0×0000. If the checksum is not set to
and multicast packets. 0×0000 by the source VTEP, then the Reserved – A set of fields, 24 bits and
Ethertype – This is set to 0×0800 as the receiving VTEP should verify the 8 bits, that are reserved and set to
payload packet is an IPv4 packet. IPv6 is Fragmented packets are dropped by the checksum and if not correct, the frame zero
planned. receiver. must be dropped and not decapsulated.

58 Copyright © 2014 Juniper Networks, Inc.

 VXLAN- Why UDP
• VXLAN uses UDP encapsulation to take advantage of the load
balancing in the network.
• The UDP source port is set to the hash of inner packet fields and the
UDP destination port is set to the 4789
• Setting the UDP source port as packet hash allows for load balancing
of the packets using 5-tuples.
• The existing IP network infrastructure supports this and no changes
are required to support VXLAN in the network

59 Copyright © 2014 Juniper Networks, Inc.

 New Config knobs in 15.1X53-D60
• set protocols evpn default-gateway no-gateway-community

If each PE has the same MAC address configured on the IRB interface, there is no
need to dynamically synchronize the IRB MACs through the control plane by
advertising them with the default gateway extended community

• set interfaces irb unit 0 proxy-macip-advertisement

In case of Non Collapsed, This config is required on Spine(L3GW) to advertise MAC+IP
route

• set vlans <> vxlan ingress-node-replication is optional.

60 Copyright © 2014 Juniper Networks, Inc.

 Consideration for 15.1X53-D60
• If the following config exist, then It should be removed and device need to be rebooted
set routing-options forwarding-table no-indirect-next-hop
set routing-options forwarding-table no-indirect-next-hop-change-acknowledgements

61 Copyright © 2014 Juniper Networks, Inc.

 Ingress-node-replication
Fabric
"set vlans <> vxlan ingress-node-replication" is optional.

Recommendation is not to configure the knob.

Spine-1 Spine-2

With Knob:
Leaf-1>show vlans bd1

Routing instance VLAN name Tag Interfaces

default-switch bd1 1
et-0/0/67.0
vtep.32769*
vtep.32770*
VNID : 100 VNID : 100 VNID : 200
Leaf-1 Leaf-2 Leaf-3
Without Knob:
Leaf-1>show vlans bd1

Routing instance VLAN name Tag Interfaces

default-switch bd1 1
et-0/0/67.0
vtep.32769*

VM1 VM2 VM3

VLAN : 10 VLAN : 10 VLAN : 20

62 Copyright © 2014 Juniper Networks, Inc.

 ECMP
• QFX5100 :
- Per VNI only one VTEP will be chosen
- ECMP underlay to VTEP endpoint works

• QFX10K :
- ECMP across VTEP works
- ECMP underlay to each VTEP endpoint works

63 Copyright © 2014 Juniper Networks, Inc.