Cisco Live Fabricpath

BRKDCT-2202
FabricPath Migration Use Cases

BRKDCT-2202
FabricPath Migration Use Cases
Azeem Suleman, Solutions Architect

Mike Murphy, Solutions Architect
BRKDCT - 2202 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Goal
After this session you will be acquainted with the following:
Grasp key concepts and caveats when deploying FabricPath

Understand deployment use cases with HW & SW dependencies
Walk through migration process, step by step plan for select use
cases
Commands to verify the migration to FabricPath
BRKDCT - 2202 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public3 3
Follow On Sessions
Session suggestions
BRKARC-2470: Cisco Nexus 7000 Switch Architecture

BRKARC-3452: Cisco Nexus 5500 and 2000 Switch Architecture
BRKDCT-2081: Cisco FabricPath Technology and Design**
Visit the Design Clinics
Meet the Engineer & Advanced Services Booth’s

Open 10 – 6 Daily
BRKDCT - 2202 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public4 4
Agenda
1. FabricPath Overview, pre migration items
 HW & SW Dependencies
2. Migration Use cases

 Use cases overview
 Migration walk through
3. Troubleshooting commands
4. Q&A
5. References
BRKDCT - 2202 © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
1. FabricPath Overview, dependencies
Goal’s of Cisco FabricPath
When Deployed across multiple Cisco Nexus chassis', FabricPath creates a flat data center switching fabric
Switching Routing
 Easy Configuration  Multi-pathing (ECMP)
 Plug & Play  High cross sectional bandwidth
 Provisioning Flexibility  Fast Convergence\Low latency
 Highly Scalable
FabricPath
“FabricPath brings Layer 3 routing benefits to

flexible Layer 2 bridged Ethernet networks”
FabricPath : Supported Hardware & Software
For Your Platform NX-OS version

Reference
Minimum Recommended
Nexus 7000 with F1 5.2.3(a) 5.2(7)
Nexus 7000 with F2 6.0.2 6.1(4)
Nexus 7000 with F2e 6.1(2) 6.1(4)
Nexus 5500 !, *No 5000’s 5.1(3)N2(1c) 5.2(1)N1(4)
Nexus 6000 6.0(2)N1(1a) 6.0(2)N1(2)
Reason to recommend 5.2(7) is that 5.2 train is the long live release
If not explicit mention, the synonym F2 for N7k means the F2 as well F2e linecards
Nexus 5500 Minimum Recommended Cisco NX-OS Releases
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/release/recommended_releases/recommended_nx-os_releases.html
http://www.cisco.com/en/US/docs/switches/datacenter/nexus6000/sw/release/recommended_releases/recommended_nx-os_releases.html
http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/recommended_releases/recommended_nx-os_releases.html
FabricPath & CE Vlans * Key Points !
 2 types of vlans
 CE (Classic Ethernet, default) FabricPath
 FabricPath (FP)
 FP vlans cannot go on M1,M2 modules *7K
 Only FP vlans will be carried over FP interfaces
Classic Ethernet
 FP vlans can be mixed with CE vlans on edge interfaces
Port Type VLANs allowed to be configured VLANs allowed to be brought up
N7K-M1, N7K-M2 FP, CE CE

Edge == switchport mode
N7K-F1, N7K-F2 Edge FP, CE FP, CE access || trunk
N7K-F1, N7K-F2 Core FP, CE FP Core == switchport mode
N6000, Core\Edge FP, CE FP, CE fabricpath
N5500, Edge FP, CE FP, CE
N5500, Core FP, CE FP
FabricPath Terminology
 Send/receive FabricPath frame

 No STP, no MAC learning, no MAC address table
FP Core Ports  Utilizes a routing table computed by IS-IS
S10 S20 S30 S40
Spine Switch
FabricPath (FP)
S100 S200 S300
Leaf Switch
1/1 1/2
Classical Ethernet (CE)
A B
 Send/receive regular Ethernet frames

CE Edge Ports  Running STP, does MAC address learning using MAC address table
2. FabricPath Use Cases overview
11
FabricPath Use cases
CASE SPINE LEAF
1 Nexus 7000 with M1/F1 Nexus 5500

Or N6004 Nexus 6000
2 Nexus 7000 with F2 Nexus 7000 with M1/F1 cards
3 Nexus 7000 with F2 Nexus 5500

N6004 Nexus 6000
4 Nexus 7000 with M1/F1 Nexus 5500/6000 and Nexus 7000 with
F2 card
Fabric Path Use case 1
 Nexus 7000\6000 as spine
FabricPath (FP)
‒ F1/M1 line card combination
Spine
 Nexus 5500\6000 as Leaf
M1,F1
‒ Dual homed FEXes with EvPC
‒ Double sided vPC
 Peer-link initially built on M1 Port
on N7K
‒ Migrated to F1 Port on N7K Leaf
 Design description
‒ Layer 3 termination in spine
‒ Network Service appliance
connected to the spines via vPC
FabricPath Use case 2
 Easy migration from traditional setup Classical Ethernet (CE)
 All VLAN(s) available everywhere FabricPath (FP)
 Link capacity between Spine and Leaf can Leaf M1,F1
easily increased
 No Spanning Tree between Spine and Leaf
 Legacy Layer 2 switches can be connected
without Spanning Tree blocked ports F2
Spine
 End devices can be connected as
active/active
 Future options:
‒ Mix Chassis with F2e and M1 / M2 linecards to
increase L2 and L3 scalability and combine Leaf
features of F2e and M cards
‒ Small to medium recommendation; use Nexus
6004 in spine
Nexus 7000\6000 as Spine FabricPath (FP)
- F2 line card
Nexus 5500 or 6000 as Leaf

F2
Spine
Layer 2 design with double sided vPC
Peer-links
-F2 Ports at Spine Layer
Leaf
 Nexus 7000 as Spine FabricPath (FP)
‒ F1 line card
M1,F1
Spine
 Nexus 5500 / Nexus 7000 F2 as Leaf
 Dual Homed FEXes with EvPC F2

Leaf
 Layer 2 design with double sided vPC
 Peer-link built on F1 / F2 Port

‒ Between N5K(s) Leaf
Pure Leaf Use Case * Brief Example
 Nexus 5500 as Pure Leaf
 Design description
Backup
‒ For small data centers Backup
LISTEN
LISTEN
‒ With Layer 3 up to 16 FEX per fire-cell
vPC+
‒ Servers distributed in two fire-cells
‒ Running Layer 3 and Layer 2 on all 4 Nexus 5500 vPC+
FTAG-1 FTAG-2
‒ HSRP active on 2 Nexus 5500, listen / listen in second
ACTIVE ACTIVE
Data Center
Other option is HSRP active on all 4 Nexus 5500’s by
different HSRP passwords and vPC+
With this option, no other devices connected via
FabricPath are allowed
‒ Network Service appliance could be connected via vPC
‒ Connect end-devices via vPC to the Leaf switches
Benefits Pure Leaf
 Even with only 4 active devices, a high

number of ports can be achieved
 Fast convergence by FabricPath Backup
Backup
LISTEN
 All VLAN(s) available everywhere in LISTEN
vPC+
topology
 Flexible growth of throughput between
vPC+
fire-cells using a FabricPath PortChannel FTAG-1 FTAG-2
 Future Options ACTIVE ACTIVE
‒ Using FabricPath “anycast HSRP” to

allow all 4 Nexus 5500 doing Layer 3
forwarding
‒ Running Nexus 6004 as Pure Leaf to
expand number of FEX and servers Focus of this setup is,
small and flexible PODs
Spine Use Case
 Nexus 7000 as spine Spine
‒ With F1 or F2 linecards
 Nexus 5500 as leaf

 Nexus 7000 as border leaf
‒ With F2 linecards
 Design description FTAG-1 FTAG-2
‒ Seen this more in recent designs Leaf ACTIVE ACTIVE

vPC+
‒ Spine is Layer 2 only
vPC+ Layer 2
‒ Safe side up to 16k MAC, 2k VLAN
‒ Using learning per SOC higher MAC scale can
Layer 3
be reached
‒ Layer 3 termination in Border Leaf
‒ HSRP active / active, by vPC+ both spines do
Layer 3 forwarding
‒ Network Service appliance connected to Border Border Leaf
Leaf
‒ Connect end-devices via vPC to the Leaf
switches
Benefits Spine
 High Availability Spine
 Very scalable in terms of amount of
VLANs and throughput due to
FabricPath
 No MAC learning in Spine
 MAC scale on Border Leaf is up to 16k,
FTAG-1 FTAG-2
or with learning per SOC higher
ACTIVE ACTIVE
 The network services can be Leaf vPC+
concentrated in one or more Border vPC+ Layer 2
Leafs
‒ E.g. in second Data Center
Layer 3
 Future solutions:
‒ Run at Border Leaf in the Nexus 7000 chassis
F2e with M cards to enable more features and
higher scalability
‒ F / M card interaction max 80Gbps for Layer 3, Focus of this setup is, Border Leaf
no issue for F2 only high Availability and high scalability
‒ Run in Border Leaf Nexus 6004
FabricPath DCi Use Case
 Nexus 7000\6000 as spine Data Center 2
‒ With F2
vPC+
Leaf DC-2
 Nexus 5500 as Leaf
Layer 2
 Design description Backup Backup

Spine DC-2 vPC+
‒ Basis is use case “Classical” Layer 2 & 3 LISTEN LISTEN
‒ Most designs including second DC and using

FabricPath as DCi over DWDM
‒ Running vPC+ on Spines and Leafs
‒ All 4 Spines should be active router Spine DC-1 FTAG-1 FTAG-2
Layer 2 & 3 vPC+
‒ Spines in DC-2 should have higher MDT prio ACTIVE ACTIVE
then the default as “backup FTAG roots”
‒ HSRP setup is active / active and listen / listen
per VLAN, no HSRP filtering possible
Leaf DC-1 vPC+
‒ Network Services should be connected to
Layer 2
spines via vPC, distributed over both DC
‒ End devices should be connected via vPC to
the Leaf
Data Center 1
Benefits of FabricPath at the Dci, Implementation
 Benefits:
Data Center 2
‒ Very fast convergence due to FabricPath vPC+
Leaf DC-2
‒ Very scalable in amount of VLANs as well Layer 2
MAC addresses
 How to Use FabricPath as DCi: Backup Backup

vPC+
Spine DC-2
‒ Mainly can be used where today vPC or Layer 2 & 3 LISTEN LISTEN
VSS is running as DCi via DWDM
‒ Good to interconnect two Data Centers
‒ Consider to use storm-control on DCi links
‒ Current deployment observed distance is up Spine DC-1 FTAG-1 FTAG-2 vPC+
to 25km via DWDM system Layer 2 & 3
ACTIVE ACTIVE
‒ Both environments will share same multi-
destination tree because it is one Fabric
Decouple Fabric is possible by use of OTV
Leaf DC-1 vPC+
or VPLS as DCi technology
Layer 2
‒ FTAG priority should been set in all 4 spines
‒ In Future use of “anycast HSRP” would
allow all 4 Spines forwarding Layer 3 traffic Data Center 1
Layer 3/Service with >2 Layer 3 Devices
 Trying to adapt the existing model…
100-200 300-400 100-400 100-400
GLBP Anycast FHRP

Split VLANs
• Host is pinned to a • All active
• Some polarization
single gateway • Available in CY13
• Inter-VLAN traffic can
• Less granular load • Might not be available
be suboptimal
balancing soon for service
modules/appliances
2.1 Migration Use Cases
24
Migration Tips
25
Port selection with FabricPath on Nexus 7000
Allowed Disallowed
Description Encapsulation
ports ports
FabricPath core port FP F1,F2 M
vPC+ peer-link FP F1,F2 M
CE (Classical Ethernet) interface CE M,F1,F2 -
FP (FabricPath) interface FP F1,F2 M

Routed uplinks N/A M,F2 -
Note:
If not explicit mention, the synonym F2 for N7k means the F2 as well the F2e linecards
If switch is part of the FabricPath domain, then edge port has to be F1 or F2 port
It can not be M1 port (since it can not work with FabricPath VLANs)
vPC+ can be formed with the same type of port only (no mix of F1 and F2, M1 is not allowed at all)
VLAN mode selection with FabricPath
Interface Cross
Description Ports Comment
mode FabricPath
CE VLAN CE M1,F1,F2 ✗ Valid design
CE VLAN FP F1,F2 ✗ Not valid design
FP VLAN CE F1,F2 ✓ Valid design

FP VLAN FP F1,F2 ✓ Valid design
Note:
M1 port:
- can not be access port for FabricPath VLAN
- can not be a trunk with one of the FabricPath VLAN
F1 / F2 port in FabricPath mode:

- can not be a trunk with one of the CE VLANs (CE VLAN does not cross FabricPath domain)
FabricPath and Spanning-tree
Spanning-tree root assignment
FabricPath edge ports MUST BE

spanning-tree ROOT FabricPath
CE CE
If you have not configured this port will
be in UP state, but traffic will not go Spanning-tree Domain 1 Spanning-tree Domain 2
through because all VLANs would be
blocked by spanning-tree
Each FP edge switch must be configured

as root for all FP Vlans
FP edge switches connect to the same
STP domain should have the same
bridge priority
2.2 FabricPath migration walk through
29
Case 1: Migration to FabricPath FabricPath (FP)
Standard aggregation layer vPC pair with multiple access layer vPC pairs
M1,F1
 Most common design
 M1/F1 line card combination
 Layer 2 design with double-sided vPC
 Dual homed FEXes
 Peer-Link built on M1 ports CE
 Migration steps
M1,F1
– Move peer-link to F1 ports
– Add basic non disruptive fabric-path configuration
– Convert peer-link to FabricPath, vPC domain to vPC+
domain and change VLANs to FabricPath mode.
FP
Case 1 - Steps For Migration to FabricPath
31
Case 1: Migration to FabricPath
Step 1 : vPC Peer-link from M1 to F1 Ports Classical Ethernet (CE)
FabricPath (FP)
vPC F1
M1 Peer-Link M1
F1
F1 F1 F1 F1
vPC Peer-Link Peer-Link vPC
Traffic / Host(s) Traffic / Host(s)
Case 1: Migrate peer-link from M1 to F1 ports
Step 1 : vPC Peer-link from M1 to F1 Ports
 This step is required , since M1 port can not be FP port M1,F1
 Apply script on secondary peer and on primary peer

 What is going to happen:
– Peer link goes down
– Secondary peer will shut down all SVI interfaces
– Secondary peer will shut down all vPC ports
– Peer-link will stay down for 20 seconds
– SVI ports will stay down for another 10 seconds
– vPC ports will stay down for another 30 second
 Peer pair will get back to stable in Approx.** 60-65 seconds
Case 1: Monitoring peer-link changes
Step 1: Verification Commands / Logs M1,F1
/* output to look for (on Nexus 7000)
N7K1-1 %VPC-3-VPC_PEER_LINK_DOWN: VPC Peer-link is down

N7K1-1 %VPC-5-VPC_PEER_LINK_UP: vPC Peer-link is up
…
N7K1-1 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1,10-19,100,200 on Interface port-channel1 are
removed from suspended state.
N7K1-1 %VPC-5-VPC_RESTORE_TIMER_START: vPC restore timer started to reinit vPCs
N7K1-1 %VPC-5-VPC_RESTORE_TIMER_EXPIRED: vPC restore timer expired, reiniting vPCs
Case 1: Verify peer-link changes FabricPath (FP)
Step 1: Verification commands / Logs

M1,F1
/* command to verify peer-link changes

show vpc
vPC domain : 11
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
...
vPC Peer-link status
Id Port Status Active vlans
1 Po1 up 1,10-19,100,200,1010-1599
vPC status
...
112 Po112 up success success ... 1,10-19,100,200,1010-1599
134 Po134 up success success ... 1,10-19,100,200,1010-1599
Case 1: Enabling basic FabricPath
Step 2: Enable the basic FabricPath configuration
 Need FabricPath license (Enhanced Layer 2)

 Install feature-set
 Enable fabricpath
 Configure switch-id (if not then switch id will be generated). It is recommended to use
one or two digit switch id for the physical switches and four digit switch id for the
virtual switches
 To assign root switch for the multi destination tree configure root-priority
 Fabricpath edge MUST be spanning-tree root for attached spanning-tree domains
Case 1: Verify basic FabricPath
Step 2: Verification commands – show fabricpath switch-id
/* command to verify changes
show fabricpath switch-id
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED
----------+----------------+------------+-----------+--------------------
*1 001b.54c2.1cc1 Primary Confirmed Yes No
Total Switch-ids: 1
Step 3: Spine configured with vPC+ and Access with vPC
 Peer-linkconfigured as  Peer-linkconfigured as Classical Ethernet (CE)
fabricpath interface fabricpath interface
 Include FabricPath switch-id into  Include FabricPath switch-id
FabricPath (FP)
vPC domain into vPC domain
 Change respective vlan to  Change respective vlan to
fabricpath mode fabricpath mode
vPC+
vPC F1 Peer-Link F1
 vPC to vPC+ bring FabricPat
down the vPC member
port-channel approx F1 F1 h Domain F1 F1 M1,F1
~75 Sec
vPC Peer-Link Peer-Link vPC
Case 1: Migrate to FabricPath (spine)
Step 3: Spine configured with vPC+ and Access with vPC Classical Ethernet (CE)
FabricPath (FP)
 Apply script on secondary spine peer and on primary spine peer
 What is going to happen:
– Peer link will be configured as fabricpath interface
M1,F1
– vPC domain will be changed to vPC+ domain by adding fabricpath

switch-id (this step will flap ALL vPCs)
– Change VLAN mode to fabricpath
– All spine vPCs will be down for about 75 seconds
 After change we will have vPC on the access side and vPC+ on the
spine side.
 Only peer-link will have interfaces configured in fabricpath mode
 Verify configuration with show fabricpath switch-id command
Case 1: Monitoring changes on spine
Step 3: Monitoring (1) M1,F1

N7K1-1 %VPC-3-VPC_PEER_LINK_DOWN: VPC Peer-link is down
N7K1-1 %ETHPORT-5-IF_UP: Interface port-channel1 is up in mode fabricpath
N7K1-1 %VPC-5-VPC_PEER_LINK_UP: vPC Peer-link is up
N7K1-1 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1,10-19,100,200,1010-1599 on Interface port-
channel1 are removed from suspended state.
N7K1-1 %VPC-5-VPC_RESTORE_TIMER_EXPIRED: vPC restore timer expired, reiniting vPCs
N7K1-1 %VPC-5-VPC_UP: vPC 112 is up
N7K1-1 %VPC-5-VPC_UP: vPC 134 is up
Case 1: Monitoring changes on spine
Step 3: Monitoring (2) FabricPath (FP)
M1,F1

N5548UP-1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel112 is
down (No operational members)
N5548UP-1 %ETHPORT-5-IF_UP: Interface port-channel112 is up in mode trunk
Case 1: Verify changes on spine
Step 3: Verification Commands – show vpc
FabricPath (FP)
M1,F1

show vpc
vPC domain id : 11
vPC+ switch id : 11
vPC fabricpath status : peer is reachable through fabricpath
Step 3: Verification Commands – show fabricpath switch-id
FabricPath (FP)
/* output to look for (on Nexus 7000) M1,F1
=========================================================================
----------+----------------+------------+-----------+--------------------
2 001b.54c2.1e41 Primary Confirmed Yes No
1011 001b.54c2.1e41 Primary Confirmed No Yes
1011 001b.54c2.1cc1 Primary Confirmed No Yes
Total Switch-ids: 4
Step 3: Verification commands – show fabricpath isis adjacency

show fabricpath isis adjacency
Fabricpath IS-IS domain: default Fabricpath IS-IS adjacency database:
System ID SNPA Level State Hold Time Interface
N7K2-1 N/A 1 UP 00:00:29 port-channel1
Step 3: Verification commands – show fabricpath route
show fabricpath route Topology (ftag), Switch
ID,
/* output to look for (on Nexus 7000) Sub-Switch ID
FabricPath Unicast Route Table
'a/b/c' denotes ftag/switch-id/subswitch-id
'[x/y]' denotes [admin distance/metric]
ftag 0 is local ftag Administrative distance,
subswitch-id 0 is default subswitch-id routing metric
FabricPath Unicast Route Table for Topology-Default

Route age
0/1/0, number of next-hops: 0
via ---- , [60/0], 7 day/s 00:53:24, local
0/1011/11, number of next-hops: 1 Next-hop interface(s)
via Po112, [80/0], 7 day/s 00:50:28, vpcm
via Po134, [80/0], 7 day/s 00:50:28, vpcm
via Po1, [115/20], 0 day/s 00:24:18, isis_fabricpath-default
via ---- , [60/0], 7 day/s 00:50:23, local Client protocol
via ---- , [60/0], 7 day/s 00:50:23, local
Case 1: Verify spanning-tree
Step 3: Verification commands – show spanning-tree vlan 100 (1) Classical Ethernet (CE)
/* command to verify changes FabricPath (FP)
show spanning-tree vlan 100

M1,F1

VLAN0100
Spanning tree enabled protocol rstp
Root ID Priority 4196
Address c84c.75fa.6000
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 4196 (priority 4096 sys-id-ext 100)

Address c84c.75fa.6000 Note:
Nexus 7000 spanning-tree
Interface Role Sts Cost Prio.Nbr Type priority is 4096.
---------------- ---- --- --------- -------- --------------------------------
Po112 Desg FWD 1 128.4207 (vPC) P2p
Step 3: Verification commands – show spanning-tree vlan 100 (2)
FabricPath (FP)
show spanning-tree vlan 100 M1,F1

VLAN0100
Cost 2
Port 4096 (port-channel1)

Address 547f.ee52.87bc
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec Note:
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- -------------------------------- Nexus 7000 spanning-tree
Po1 Root FWD 1 128.4096 (vPC peer-link) Network P2p priority is 4096.
Po112 Root FWD 1 128.4207 (vPC) P2p
Eth101/1/1 Desg FWD 1 128.1153 (vPC) Edge P2p
Step 3: Verification commands – show spanning-tree vlan 100 (3) Classical Ethernet (CE)
FabricPath (FP)

M1,F1
show spanning-tree vlan 100
VLAN0100
This bridge is the root
Note:
Address c84c.75fa.6000 Nexus 7000 spanning-tree
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec priority is 32868.
Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- --------------------------------
Po112 Desg BKN*1 128.4207 (vPC) P2p *L2GW_Inc
…
N7K1-1 %STP-2-L2GW_BACKBONE_BLOCK: L2 Gateway Backbone port inconsistency blocking port port-channel112 on
VLAN100.
Step 4: adding leaf to FabricPath Domain  Remove vPC Classical Ethernet (CE)
member port
 Peer-link  Remove vPC
 Configure member FabricPath (FP)
configured as member port
port in Fabricpath
fabricpath  Configure member
interface port in Fabricpath
 Include vPC+
FabricPath
switch-id into M1,F1
vPC domain F1 Peer-Link F1
 Change
respective
FabricPath
vlan to F1 F1 Domain F1 F1
fabricpath
mode
 Peer-link
configured as vPC+
fabricpath Peer-Link Peer-Link vPC
interface
 Include
FabricPath
switch-id into
vPC domain
 Change
respective vlan
to fabricpath Traffic / Host(s) Traffic / Host(s)
mode
Case 1: Migrate to FabricPath (leaf)
Step 4: Configure FabricPath leaf
FabricPath (FP)
 Apply script on secondary leaf peer and on primary leaf peer

M1,F1
 Apply script on secondary spine peer and on primary spine peer
 Following will be changed in configuration:
– vPC domain will be changed to vPC+ domain by adding fabricpath switch-id
– Remove vPC member interfaces from Port-Channel and configure it in fabricpath mode on the
leaf switches and on the spine switches
– Connectivity between spine and leaf will be down for about 40 seconds for the peer-link and
fabricpath links (spine to leaf), up to 3 minutes for the FEX vPC.
 After change we will have fabricpath domain formed with spine switches and pair of
access switches.
 Verify configuration
Case 1: Monitoring changes on leaf
Step 4: Verification commands FabricPath (FP)
M1,F1

N7K1-1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel112: Ethernet3/1 is down
…
N7K1-1 %ETHPORT-5-IF_UP: Interface Ethernet3/1 is up in mode fabricpath
N7K1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet3/1 up in fabricpath topology 0
N7K1-1 %ETHPORT-5-IF_UP: Interface Ethernet3/2 is up in mode fabricpath
N7K1-1 %L3VM-5-FP_TPG_INTF_UP: Interface Ethernet3/2 up in fabricpath topology 0
Case 1: Monitoring changes on leaf
Step 4: Verification commands FabricPath (FP)
M1,F1
N5548UP-1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel1: port-channel1 is down
N5548UP-1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel112: port-channel112 is down
N5548UP-1 %PFMA-2-FEX_STATUS: Fex 101 is offline
N5548UP-1 %PFMA-2-FEX_STATUS: Fex 102 is offline
N5548UP-1 %ETHPORT-5-IF_UP: Interface port-channel1 is up in mode fabricpath
N5548UP-1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode fabricpath
N5548UP-1 %ETHPORT-5-IF_UP: Interface Ethernet1/2 is up in mode fabricpath
N5548UP-1 %ETHPORT-3-IF_ERROR_VLANS_REMOVED: VLANs 1,10-19,100,200,1010-1599 on
Interface port-channel1 are removed from suspended state.
N5548UP-1 %PFMA-2-FEX_STATUS: Fex 102 is online
N5548UP-1 %PFMA-2-FEX_STATUS: Fex 101 is online
Case 1: Verify changes on leaf
Step 4: Verification Commands - show fabricpath switch-id Classical Ethernet (CE)
/* command to verify changes FabricPath (FP)

M1,F1
FABRICPATH SWITCH-ID TABLE
Legend: '*' - this system
=========================================================================
-----------+----------------+-----------+------------+-------------------
1 001b.54c2.1cc1 Primary Confirmed Yes No
*11 547f.ee52.87bc Primary Confirmed Yes No
12 547f.ee29.4ec1 Primary Confirmed Yes No
13 547f.ee24.4381 Primary Confirmed Yes No
14 547f.ee04.023c Primary Confirmed Yes No
1112 547f.ee29.4ec1 Primary Confirmed No Yes
1112 547f.ee52.87bc Primary Confirmed No Yes
1134 547f.ee24.4381 Primary Confirmed No Yes
1134 547f.ee04.023c Primary Confirmed No Yes
Total Switch-ids: 12
Step 4: Verification Commands - show fabricpath route (1) Classical Ethernet (CE)
FabricPath (FP)
show fabricpath route M1,F1

ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

via ---- , [60/0], 1 day/s 04:10:49, local
via Eth1/1, [115/40], 0 day/s 05:33:25, isis_fabricpath-default
Step 4: Verification Commands - show fabricpath route (2)
FabricPath (FP)

via Eth1/2, [115/40], 0 day/s 05:33:25, isis_fabricpath-default M1,F1
via ---- , [60/0], 0 day/s 05:33:20, local
via ---- , [60/0], 0 day/s 05:33:20, local
Step 4: Verification Commands – show vpc (1)
FabricPath (FP)
M1,F1
show vpc
Legend: (*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 112
vPC+ switch id : 112
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 67
Peer Gateway : Enabled
Peer gateway excluded VLANs : -
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Case 1: Verify changes on leaf FabricPath (FP)
M1,F1
vPC Peer-link status

---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up 1,10-19,100,200,1010-1599
vPC status
---------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans vPC+ Attrib
-- ---------- ------ ----------- ------ ------------ -----------
101 Po101 up success success - DF: Partial
112 Po112 down* success success - DF: No
Step 5: FabricPath domain configuration for other leaf nodes

 Remove vPC member  Remove vPC member
port port
 Configure member  Configure member port
 Peer-link configured
port in FabricPath in FabricPath as fabricpath
interface
 Include FabricPath
vPC+ switch-id into vPC
Peer-Link
domain
 Change respective
FabricPat vlan to fabricpath
h Domain mode
vPC+ Peer-Link Peer-Link vPC+

vPC
 Peer-link configured
as fabricpath
interface
switch-id into vPC
domain
 Change respective
vlan to fabricpath
Traffic / Host(s) Traffic / Host(s) mode
Case 1: Migration to FabricPath - Summary
Migration Convergence
 M1 to F1 peer-link migration (non disruptive for traffic flow)

– peer link down for 15 second
– vPC ports on secondary peer will be down for up to 65 second
– no traffic interruption (just bandwidth reduction)
 Build FabricPath spine (disruptive for all layer 2 domain)
– vPC will be down for about 75 seconds (no traffic at all)
 Add leaf to FabricPath domain (disruptive for portion of layer 2 domain)
– vPC will be down for about 40 seconds (no traffic at all) for peer-link and fabricpath links
between spine and leaf, up to 3 minutes for the dual connected FEX
Case 2 – Migrate Leaf to FabricPath
60
Aggregation layer vPC pair with multiple access layer vPC pairs M1,F1
 Adding F2 line card to the M1/F1 environment

 Layer 2 design with double-sided vPC
 Dual homed FEXes
 Peer-Link built on M1 ports CE
 Migration steps M1,F1
– Move peer-link to F1 ports

– Add basic non disruptive fabric-path configuration F2
– Convert peer-link to FabricPath, vPC domain to vPC+ domain and

change VLANs to FabricPath mode.
FP
Aggregation layer vPC pair with multiple access layer vPC pairs (2) M1,F1
– Add F2 card to the chassis

– Create dedicated VDC for F2
– Adding links from F2 to F1, and to Nexus 5500 CE
– Build fabricpath domain out of new F1 and F2 ports M1,F1
– Move N5K from F1 to F2

– Convert N5K peer-link to FabricPath, vPC domain to vPC+ F2
domain and change VLANs to FabricPath mode.
FP
Case 2 - Steps For Migration to FabricPath
63
Case 2: Steps common with case 1
Step1 – 3: The same as for case 1
 vPC Peer-link from M1 to F1 Ports (we will skip this step in case 2)
 Enable the basic FabricPath configuration (case 1 step 2)
 Spine configured with vPC+ and Access with vPC (case 1 step 3)
vPC+
 Peer-link configured as
 Peer-link F1 Peer-Link F1
configured as fabricpath interface
FabricPath  Include FabricPath
fabricpath
interface Domain F1
switch-id into vPC
F1 F1 F1
domain
 Change respective vlan
switch-id into vPC
domain to fabricpath mode
 Change respective F2 F2 F2 F2
vlan to fabricpath
mode F2 Peer-Link F2
 Peer-link
F2 F2 configured as
F2 F2
 Peer-link fabricpath
configured as interface
fabricpath  Include FabricPath
interface switch-id into vPC
 Include domain
Peer-Link Peer-Link
FabricPath  Change respective

switch-id into
vPC
vPC+ vPC
vPC+ vlan to fabricpath
vPC domain mode
 Change
respective vlan to
fabricpath mode
Case 2: Migrate to FabricPath (spine)
Step 4: Configure dual layer spine M1,F1
 Configure VDC for F2 on mixed chassis.

 Configure F2 VDC for fabricpath (enable fabricpath, configure F2
interswitch links, uplinks and downlinks for fabricpath)

 Configure F1 interfaces connected to F2 as fabricpath
Case 2: Verify changes on spine M1,F1
Step 4: Verification Commands - show fabricpath switch-id F2

=========================================================================
-----------+----------------+-----------+------------+-------------------
Total Switch-ids: 6
Case 2: Verify changes on spine M1,F1
Step 4: Verification Commands - show fabricpath isis adjacency F2

N7K1-1 N/A 1 UP 00:00:28 Ethernet8/5
N7K2-1 N/A 1 UP 00:00:30 Ethernet8/6
N7K2-3 N/A 1 UP 00:00:27 Ethernet8/7
N7K2-3 N/A 1 UP 00:00:29 Ethernet8/8
Case 2: Migrate to FabricPath (leaf) M1,F1
Step 5: Move leaf connections to the F2 spine F2
 Configure leaf switches for fabricpath

(disable uplinks to F1 and enable uplinks to F2)
 Following will be changed in configuration (on the leaf):
– vPC domain will be changed to vPC+ domain by adding fabricpath switch-id
– Remove vPC member interfaces from Port-Channel and configure it in fabricpath mode on
the leaf switches and on the spine switches
– Connectivity between spine and leaf will be down for about 40 seconds for the peer-link and
fabricpath links (spine to leaf), up to 3 minutes for the FEX vPC.
 After change we will have fabricpath domain formed with spine switches and pair
of access switches.
Step 5: Verification Commands - show fabricpath switch-id (2)
M1,F1
=========================================================================
SWITCH-ID SYSTEM-ID FLAGS STATE STATIC EMULATED F2
-----------+----------------+-----------+------------+-------------------
*11 547f.ee52.87bc Primary Confirmed Yes No
12 547f.ee29.4ec1 Primary Confirmed Yes No
13 547f.ee24.4381 Primary Confirmed Yes No
14 547f.ee04.023c Primary Confirmed Yes No
1112 547f.ee29.4ec1 Primary Confirmed No Yes
1112 547f.ee52.87bc Primary Confirmed No Yes
1134 547f.ee04.023c Primary Confirmed No Yes
1134 547f.ee24.4381 Primary Confirmed No Yes
Total Switch-ids: 14
/* command to verify changes M1,F1
show fabricpath route

/* output to look for (on Nexus 5500) F2
ftag 0 is local ftag
subswitch-id 0 is default subswitch-id

via ---- , [60/0], 10 day/s 15:32:08, local
via Po134, [80/0], 10 day/s 15:27:39, vpcm
via Eth1/10, [115/80], 0 day/s 00:20:02, isis_fabricpath-
default
Case 2: Verify changes on leaf M1,F1
1/2/0, number of next-hops: 2 F2
via ---- , [60/0], 10 day/s 15:27:28, local
via ---- , [60/0], 10 day/s 15:27:28, local
Step 5: Verification Commands – show vpc (1) F2

show vpc
Legend: (*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 134
vPC+ switch id : 134
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 67
Peer Gateway : Enabled
Peer gateway excluded VLANs : -
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
M1,F1
vPC Peer-link status F2

---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up 1,10-19,100,200,1010-1599
vPC status
---------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans vPC+ Attrib
-- ---------- ------ ----------- ------ ------------ -----------
134 Po134 down* success success - DF: No
Step 5: Verification Commands – show fabricpath isis adjacency F2

N5548UP-1 N/A 1 UP 00:00:23 Ethernet8/1
N7K1-1 N/A 1 UP 00:00:21 Ethernet8/5
N7K2-1 N/A 1 UP 00:00:28 Ethernet8/6
N7K2-3 N/A 1 UP 00:00:27 Ethernet8/7
N7K2-3 N/A 1 UP 00:00:23 Ethernet8/8
Migration results
76
FabricPath migration timing
Step in migration Time Where

M1 to F1 peer-link 65 sec spine
Basic fabricpath configuration n/a spine,leaf
vPC to vPC+, fabricpath VLANS, peer-link fabricpath 75 sec spine
vPC to vPC+, fabricpath VLANS, peer-link fabricpath - leaf
75 sec
Uplink vPC to individual fabricpath links - spine, leaf
Previous step with dual attached FEX 1.5+ * min leaf
* - actual time depends on number of FEXes

Migration to FabricPath best practices
 Start on spine (Nexus 7000 as aggregation)

 Migrate peer-link from M1 to F1 . This step might be optional if peer-link is built
on M1 ports in the first place
 On Nexus 7000 migrate access/leaf switches connection ports to F1
 Configure FabricPath basic configuration on devices migrating to FabricPath
 Convert spine with peer-link as FabricPath, VLANs in fabricpath mode, change
vPC configuration to vPC+
 Configure leaf with peer-link as FabricPath, VLANs in fabricpath mode, change
vPC configuration to vPC+, convert uplink to spine from vPC into individual
fabricpath links (on both leaf and spine)
 Repeat last step for all leaf devices
3. Troubleshooting FabricPath
79
Switch-id Conflict
Nexus# sh fabricpath switch-id

=========================================================================
----------+----------------+----------+----------+--------+--------------
*3 c062.6bac.e343 Primary Confirmed Yes No
30 547f.ee02.ce3c Primary Confirmed Yes No
40 547f.ee04.5cfc Primary Confirmed Yes No
Nexus(config-if-range)# no shut
18:46:04 %FABRICPATH-2-FABRICPATH_LINK_BRINGUP_STALLED_STATIC: Link bringup stalled due to conflicts
Nexus# sh fabricpath conflict all

====================================================
Port State
---------------+-------------------------+----------
Ethernet3/31 Suspended due to conflicts
==============================================
Fabricpath Conflicts
SYSTEM-ID SWITCH-ID STATIC
---------------+------------+---------------
c062.6bac.e343 3 Yes
c062.6bac.e342 3 Yes
Topology
Nexus(config)# fabricpath topology 1

Nexus(config-fp-topology)# member vlan 3002
Nexus(config-fp-topology)# show fabricpath topology vlan
Topo-Description Topo-ID Configured VLAN List
-------------------------------- --------- -------------------------------------
0 0 1-3001, 3003-4095
1 1 3002
Nexus(config)# int e3/25

Nexus(config-if)# fabricpath topology 1
Nexus# sh fabricpath topology interface

Interface Topo-Description Topo-ID Topo-IF-State
------------------- -------------------------------- ---------- ----------------------
-----
port-channel1 0 0 Up
Ethernet3/7 0 0 Up
Ethernet3/25 1 1 Up
FabricPath TAC Tools
 show tech fabricpath

– isis - FabricPath IS-IS information
– switch-id - Detailed Information for FabricPath switch-id module
– Topology - Detailed Information for topology troubleshooting
 These do not include FabricPath routes, MAC or comprehensive forwarding

related information
FabricPath Troubleshooting Commands
 show feature-set - check if fabricpath is enabled

 show run fabricpath - check fabricpath configuration
 show vpc - verify peer-link mode and vpc status
 show fabricpath switch-id - list of switches in fabricpath domain
 show fabricpath isis interface [brief] - verify interfaces are UP / Forwarding
 show fabricpath isis adjacency - list of adjacent fabricpath devices
 show fabricpath route - list of layer 2 fabricpath routes
 show interface ethernet x/y capabilities - check port information
 pong dest <sw#> dest <mac-address> vlan <vlan> count <#> … [detail]
- Used for Hop-by-Hop Latency Measurements
Q&A
84
References
CISCO FabricPath white paper
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-687554.html
Cisco FabricPath Migration Guide (to be updated):

Cisco FabricPath Design Guide:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/guide_c07-690079.html For Your
Reference
Cisco white paper scale Data Center with FabricPath
CISCO Design Zone for Data Centers

http://www.cisco.com/en/US/netsol/ns743/networking_solutions_program_home.html
Nexus 5500 Configuration Limits for NX-OS 5.2(1)N1(1)

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration_limits/limits_521/nexus_5000_config_limits_
521.html
Nexus 7000 Configuration Limits for NX-OS 6.1 train

http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NX-
OS_Verified_Scalability_Guide.html
Call to Action
 Visit the Cisco Campus at the World of Solutions

to experience Cisco innovations in action
 Get hands-on experience attending one of the Walk-in Labs
 Schedule face to face meeting with one of Cisco’s engineers

at the Meet the Engineer center
 Discuss your project’s challenges at the Technical Solutions Clinics
 Mobile APP for Agenda, Surveys and Transportation
Complete Your Online Session Evaluation
 Give us your feedback and

you could win fabulous prizes.
Winners announced daily.
 Receive 20 Cisco Daily Challenge
points for each session evaluation
you complete.
 Complete your session evaluation
online now through either the
mobile app or internet kiosk Maximize your Cisco Live experience with your free
stations. Cisco Live 365 account. Download session PDFs,
view sessions on-demand and participate in live
activities throughout the year. Click the Enter Cisco
Live 365 button in your Cisco Live portal to log in.
Thank you
For Your
Reference
More commands for verification of FabricPath
Verification of FabricPath
 To verify which VLANs are in the VLAN database in active and in mode
FabricPath
Nexus7000# show fabricpath topology vlan active
Topo-Description Topo-ID Active VLAN List
-------------------------------- --------- -------------------------------------
0 0 1, 100-109
 To verify if the FabricPath switch is STP root for VLANs and if he have
CE ports, these are all in Forwarding
Nexus5500# show spanning-tree summary
Switch is in rapid-pvst mode
Root bridge for: VLAN0001, VLAN0100-VLAN0101
L2 Gateway STP bridge for: VLAN0001, VLAN0100-VLAN0101
Port Type Default is disable
...
Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ----------
VLAN0001 0 0 0 1 1
VLAN0100 0 0 0 1 1
VLAN0101 0 0 0 1 1
---------------------- -------- --------- -------- ---------- ----------
3 vlans 0 0 0 3 3
Nexus5500#
 To check if all interfaces in mode “f-path” are connected.
– Also physical interfaces in a PortChannel (Eth 1/1 and Eth 1/2) are listed
– For Eth 1/5 the SFP is missing
Nexus5500# show interface status | in f-path
Eth1/1 vPC-peer-link connected f-path full 10G 10Gbase-SR
Eth1/2 vPC-peer-link connected f-path full 10G 10Gbase-SR
…
Eth1/5 -- sfpAbsent f-path full 10G --
Po10 vPC-peerlink connected f-path full 10G --
 If in the FabricPath environments e.g. switch-ID conflicts been detected,

it can be verified with command below
Nexus7000# show fabricpath conflict all
No Fabricpath ports in a state of resource conflict.
No Switch id Conflicts
No transitions in progress
Nexus7000#
 To verify more details about the FabricPath interfaces
– Is authentication been configured for this link
– Are the default values for hello and hold timer been used
– What is the hold timer, or latest time for the next IIH
– Is the metric calculated based on reference bandwidth or static configured
Nexus7000# show fabricpath isis interface ethernet 2/3

Fabricpath IS-IS domain: default
Interface: Ethernet2/3
Status: protocol-up/link-up/admin-up
Index: 0x0001, Local Circuit ID: 0x01, Circuit Type: L1
No authentication type/keychain configured
Authentication check specified
Extended Local Circuit ID: 0x1A082000, P2P Circuit ID: 0000.0000.0000.00
Retx interval: 5, Retx throttle interval: 66 ms
LSP interval: 33 ms, MTU: 1500
P2P Adjs: 1, AdjsUp: 1, Priority 64
Hello Interval: 10, Multi: 3, Next IIH: 00:00:04
Level Adjs AdjsUp Metric CSNP Next CSNP Last LSP ID
1 1 1 40 60 00:00:37 ffff.ffff.ffff.ff-ff
Topologies enabled:
Topology Metric MetricConfig Forwarding
0 40 no UP
 On Nexus 7000 it is also possible to verify the route to a specific switch
ID
– In this example from one of the spines to the vPC+ FabricPath ID of the leaf switches, because
both leaf switches announce this FabricPath ID, we have two paths
– This is
Nexus7000# today
show not implemented
fabricpath on Nexus
route switchid 106 5500, but is planned
…
Nexus7000#
 Some outputs either shown the System-ID or the hostname, the

command below show the connection between both
Nexus7000# show fabricpath isis hostname
Fabricpath IS-IS domain: default dynamic hostname table
Level System ID Dynamic hostname
1 4055.3905.04c1 AE-N7k-3
1 4055.3927.3e41* Nexus7000
1 547f.ee35.ba41 Nexus5500
1 547f.ee39.2b41 AE-N5596-4
Nexus7000#
 To verify information about multicast in FabricPath
– This command provide updates based on local received IGMP entries, as well via FabricPath
GM-LSPs
– For each VLAN at “(*, *), Flood” an entry should exist for each connected directly connected
switch-ID
Nexus7000# show fabricpath mroute vlan 201
(vlan/201, 0.0.0.0, 239.0.201.1), uptime: 2d16h, igmp

Outgoing interface list: (count: 1)
Interface Ethernet3/48, uptime: 2d16h, igmp
(vlan/201, 0.0.0.0, 239.0.201.2), uptime: 2d16h, isis

Switch-id 337, uptime: 2d16h, isis
(vlan/201, 0.0.0.0, 239.0.201.3), uptime: 2d16h, isis

…
(vlan/201, *, *), Flood, uptime: 2d16h, isis
 To check which MAC addresses are learned via FabricPath
– The first value is the FabricPath switch-ID
– The second is the sub-switch ID used only for MAC learned behind vPC+, the third value is the
LID
– The sub-switch ID and LID is not used by Nexus 5500, so if these are empty it is very probably a
MAC learned via Nexus 5500
– A LID with 1054 points to the switch itself, here a SVI and HSRP MAC
Nexus7000-Spine# show mac address-table dynamic | exclude Eth | exclude Po | ex sup
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
---------+-----------------+--------+---------+------+----+------------------
3001 0005.73e7.4f7c dynamic 0 F F 3030.0.0
200 0005.73e7.4f7c dynamic 0 F F 3030.0.0
111 0005.73e7.4f7c dynamic 0 F F 3030.0.0
107 0050.56ac.634e dynamic 600 F F 35.11.4513
108 0050.56ac.634d dynamic 600 F F 35.11.4513
102 0000.0c9f.f001 dynamic 0 F F 12.0.1054
102 0005.73e7.4f7c dynamic 0 F F 3030.0.0
102 108c.cf16.1c43 dynamic 0 F F 2.0.1054
Nexus7000-Spine# show mac address-table dynamic | exclude Eth | exclude Po | ex sup | count
 12To “count” the amount of FabricPath learned MAC, actual value is minus 4 for headline
Nexus7000-Spine#
 Overview about the root for FTAG-1 and FTAG-2, also which interfaces
are calculated for the multi-destination trees
Nexus7000# show fabricpath isis topology summary
Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary
MT-0
Configured interfaces: Ethernet2/3 Ethernet2/4 Ethernet2/5 Ethernet2/6 port-channel10
Number of trees: 2
Tree id: 1, ftag: 1, root system: 4055.3905.04c1, 3
Tree id: 2, ftag: 2 [transit-traffic-only], root system: 4055.3927.3e41, 4
Nexus7000#
• To check which interfaces are in the FTAG tree, the below command can be used
(For each FTAG it is different).
• Here from root for FTAG-1 (switch-ID 3):
Nexus7000# show fabricpath isis trees multidestination 1
Fabricpath IS-IS domain: default
Note: The metric mentioned for multidestination tree is from the root of that tree to that switch-id
MT-0
Topology 0, Tree 1, Swid routing table
4, L1
via port-channel10, metric 50
104, L1
via Ethernet2/3, metric 40
105, L1
via Ethernet2/4, metric 40
 Here is the check from second spine, switch-ID 4:

– To verify which interfaces are part of FTAG-1, here only the PortChannel 10
– From MDT root FTAB-1, the metric is required to reach the other switch-IDs, and probably this is
done via 1 hop 10GE link
Nexus7000# show fabricpath isis trees multidestination 1

…
Topology 0, Tree 1, Swid routing table
3, L1
34, L1
104, L1
105, L1
106, L1
Nexus7000#
FabricPath Trees
 Known unicast traffic is load-balanced across equal-cost
routes
 FabricPath uses 2 loop-free trees for unknown unicast,
broadcast and multicast traffic
 2 trees are for load-balancing
 For each packet tree is selected by ingress FP switch and choice
is carried in the packet header S1
 Root of tree1 is a switch highest sysID (priority+mac) SysID 50
 Root of tree2 is a switch with 2nd highest sysID S2

R
S3
SysID 10
 Tree is a least-cost-to-the-root graph, with lower sysID used as SysID 20
Lower SysID wins
tie-breaker
 In case of Tree1 root failure both roots are reelected
Nexus# show fabricpath isis topology summary
Fabricpath IS-IS domain: default FabricPath IS-IS Topology Summary S4
SysID 30
MT-0
Configured interfaces: Ethernet4/4
• Root for Tree 1
Number of trees: 2
Tree id: 1, ftag: 1, root system: 001b.54c2.4244, 4 • Root for Tree 2
Tree id: 2, ftag: 2, root system: 001b.54c2.4243, 3
MAC address learning
 Conversational learning is
 Learning MAC addresses is not required in FabricPath disabled on L3 edge
Core as switching based on Switch ID (Q:what about switches (when SVI
FTAG?) configured & up on FP
 FP Edge switches learn local MAC addresses (behind edge vlan)
ports) conventionally
 FP Edge devices learn remote addresses (behind Core-  When M1+F1 are in the
facing ports) using conversational learning same VDC, special
handling is needed to
• For packets arriving from FP, source MAC (not outer SA!)
is learned when destination MAC of the frame is already forward packets M1FP
known on any Edge port of this switch core – this is orchestrated
by MCM (mixed chassis
 No learning from broadcasts (though existing entries will be manager)
updated)
 Normal Learning from multicasts (HSRP address, etc)
Reference: Acronym Decoder
 ACL–Access Control List  NDE–NetFlow Data Export
 ADJ–Adjacency  OIF–Output Interface
 ASIC–Application Specific Integrated Circuit  OIL–Output Interface List
 CE–Classic Ethernet  PACL–Port ACL
 CLI–Command Line Interface  PBR–Policy-Based Routing
 CMP–Connectivity Management Processor  PFC–Priority Flow Control (per-priority pause)
(lights-out)  PIM–Protocol Independent Multicast
 CoPP–Control Plane Policing  POD–Pool of Devices
 COS–Class of Service  QoS–Quality of Service
 CP–Control Processor (main CPU)  RACL–Router ACL
 DCB–Data Center Bridging  RE–Replication Engine
 DCI–Data Center Interconnect  RPF–Reverse Path Forwarding
 DSCP–Differentiated Services Code Point  SFP+–10G-capable Small-Formfactor Pluggable
 ECMP–Equal Cost Multi Path  SID–Switch ID
 EOBC–Ethernet Out-of-Band Channel  SoC–System-on-chip/switch-on-chip
 ETS–Enhanced Transmission Selection  sSID–Sub-Switch ID
 FCoE–Fiber Channel over Ethernet  STP–Spanning-Tree Protocol
 FE–Forwarding Engine  SVI–Switched Virtual Interface (VLAN interface)
 FEX–Fabric Extender (Nexus 2000 family)  TCAM–Ternary CAM
 FIB–Forwarding Information Base  TLV–Type, Length, Value
 FP–FabricPath  TRILL–Transparent Interconnection of Lots of
 FRU–Field Replaceable Unit Links
 FTAG–Forwarding Tag  uRPF–Unicast RPF
 GM-LSP–Group Membership LSP  VACL–VLAN ACL
 GRE–Generic Route Encapsulation  VDC–Virtual Device Context
 HSRP–Hot Standby Router Protocol  VOQ–Virtual Output Queuing
 IGMP–Internet Group Management Protocol  VPC–Virtual Port Channel with Classic Ethernet
 IPC–Inter Process Communication  VPC+–Virtual Port Channel with FabricPath
 IS-IS–Integrated System-to-Integrated System  VRF–Virtual Routing and Forwarding
 LED–Light Emitting Diode  VRRP–Virtual Router Redundancy Protocol
 LID–Local ID  WRED–Weighted Random Early Detection
 LOU–Logical Operation Unit  WRR–Weighted Round Robin
 LSP–Link-State PDU  XL–Refers to forwarding engine with larger FIB
 MET–Multicast Expansion Table and ACL TCAMs

Cisco Live Fabricpath

Uploaded by

Document Informationclick to expand document informationFabricpath

Document Informationclick to expand document information

Copyright:

Available Formats

Cisco Live Fabricpath

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco Live Fabricpath

Uploaded by

Copyright:

Available Formats

BRKDCT-2202

FabricPath Migration Use Cases

FabricPath Migration Use Cases

Azeem Suleman, Solutions Architect

After this session you will be acquainted with the following:

Grasp key concepts and caveats when deploying FabricPath

BRKARC-2470: Cisco Nexus 7000 Switch Architecture

Visit the Design Clinics

Meet the Engineer & Advanced Services Booth’s

2. Migration Use cases

“FabricPath brings Layer 3 routing benefits to

For Your Platform NX-OS version

Port Type VLANs allowed to be configured VLANs allowed to be brought up

N7K-M1, N7K-M2 FP, CE CE

N5500, Edge FP, CE FP, CE

N5500, Core FP, CE FP

 Send/receive FabricPath frame

S10 S20 S30 S40

 Send/receive regular Ethernet frames

CASE SPINE LEAF

1 Nexus 7000 with M1/F1 Nexus 5500

2 Nexus 7000 with F2 Nexus 7000 with M1/F1 cards

3 Nexus 7000 with F2 Nexus 5500

Nexus 5500 or 6000 as Leaf

 Nexus 5500 / Nexus 7000 F2 as Leaf

 Dual Homed FEXes with EvPC F2

 Layer 2 design with double sided vPC

 Peer-link built on F1 / F2 Port

 Even with only 4 active devices, a high

 Future Options ACTIVE ACTIVE

‒ Using FabricPath “anycast HSRP” to

 Nexus 5500 as leaf

 Design description FTAG-1 FTAG-2

‒ Seen this more in recent designs Leaf ACTIVE ACTIVE

 Design description Backup Backup

‒ Most designs including second DC and using

 How to Use FabricPath as DCi: Backup Backup

GLBP Anycast FHRP

FP (FabricPath) interface FP F1,F2 M

CE VLAN FP F1,F2 ✗ Not valid design

FP VLAN CE F1,F2 ✓ Valid design

F1 / F2 port in FabricPath mode:

FabricPath edge ports MUST BE

Each FP edge switch must be configured

vPC Peer-Link Peer-Link vPC

Traffic / Host(s) Traffic / Host(s)

 This step is required , since M1 port can not be FP port M1,F1

 Apply script on secondary peer and on primary peer

/* output to look for (on Nexus 7000)

N7K1-1 %VPC-3-VPC_PEER_LINK_DOWN: VPC Peer-link is down

Case 1: Verify peer-link changes FabricPath (FP)

Step 1: Verification commands / Logs

/* command to verify peer-link changes

 Need FabricPath license (Enhanced Layer 2)

vPC Peer-Link Peer-Link vPC

Traffic / Host(s) Traffic / Host(s)

– vPC domain will be changed to vPC+ domain by adding fabricpath

/* output to look for (on Nexus 7000)