Technical whitepaper

Dual-Hop FCoE with HP Virtual Connect

modules
Cookbook

Table of contents
Introduction 3
Requirements and support 3
Virtual Connect 3
Upstream Switch 4
FlexFabric CNAs 4
Storage Array 4
Virtual Connect Limitations 5
Dual-Hop FCoE with Nexus 5xxx Series
ToR switch in FCF mode scenario 8
Requirements 9
Guidelines 9
Nexus configuration when using a
single VC to Nexus link 10
Nexus configuration when using a port
channel between VC and Nexus 13
Virtual Connect configuration 19
Converged Shared Uplink Sets (SUS)
Details and Restrictions 20
Defining two Converged Shared Uplink
Sets 21
QoS Configuration 25
Quality of Service configuration on the
Nexus 5xxx series 25
Quality of Service configuration on the
Virtual Connect modules 28
Nexus Troubleshooting commands 33
Useful Show commands 33
Per-Priority Flow Control commands 37
Issues with FCoE performance 38
Interface Status commands 43
FCoE Frames Analysis 44
Support and Other Resources 47
Contacting HP 47
Related documentation 48

2
Introduction
Dual-Hop FCoE support is one of the main features of VC 4.01. This new feature allows the FCoE traffic to be propagated
out of the enclosure to an external bridge which will handle the conversion of FCoE to FC traffic.
Dual-Hop FCoE solution has several advantages when compared to a traditional FC + Ethernet design. With FCoE, the
number of cable is reduced, this results in a simplification of the in-rack cable installation and consequently eliminates
expensive equipment and reduces the overall solution cost.
This feature is described as FCoE Dual Hop because there is two FCoE ‘hops’ between the server and the storage – the VC
Module and the external FCoE ->FC bridge that connects the HP Virtual Connect modules to the storage. No additional
external bridges are currently allowed in this configuration in order for Virtual Connect to guarantee the lossless of the
FCoE connection.

Requirements and support

Virtual Connect
Dual-Hop FCoE with HP Virtual Connect is currently only supported with the following modules:
 HP Virtual Connect FlexFabric 10Gb/24-port Module
 HP Virtual Connect Flex-10/10D Ethernet Module

Firmware requirement
Dual-Hop FCoE with HP Virtual Connect is supported with VC 4.01 or later.

FcoE-capable uplink ports

 VC FlexFabric module can only support FCoE on uplink port X1 to X4
 VC Flex-10/10D module can support FCoE on ALL uplink ports (X1-X10)

VC FlexFabric 10Gb/24-port VC Flex-10/10D

X1 – X4 X1 – X10
Uplink ports available Uplink ports available
for FCoE connection for FCoE connection

FcoE-enabled uplink port limitation

Long Range Ethernet Optical transceivers (LR SFP+) are not supported on Virtual Connect FCoE-enabled uplink port.

3
Upstream Switch
Virtual Connect Dual-Hop FCoE is currently supported only with Cisco Nexus 5000, and Cisco Nexus 5500 product
families (i.e. Nexus 5010, 5020, 5548UP and 5596UP).
Supported Nexus configuration details are available in the C-Series FCoE Switch Connectivity stream document available
on the following HP Single Point of Connectivity Knowledge (SPOCK) web page:
http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_switches.html
(Requires HP Passport; if you do not have an HP Passport account, follow the instructions on the webpage).
This SPOCK document provides detailed information on Nexus switches, recommended NX-OS versions, supported
transceivers, etc.
Nexus support
This cookbook does not intent to provide any detailed list of Cisco supported configurations, for information on Nexus
FCoE supported configuration and design, refer to the Cisco web site.

 Nexus 5000 Series Design Guides

http://www.cisco.com/en/US/products/ps9670/products_implementation_design_guides_list.html
 Data Center Design with Cisco Nexus Switches and Virtual PortChannel
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572831-
00_Dsgn_Nexus_vPC_DG.pdf

FlexFabric CNAs
SPP 2013.02 is recommended as a Baseline Release Set with the addition of Hotfix (10.70) - Linux Drivers and CNA
Firmware for VC 4.01 compatibility (includes Emulex CNA FW 4.2.401.2215 and new drivers for RHEL 5.8/5.9 - RHEL
6.2/6.3 - SLES 10 SP4 - SLES 11 SP2).
More details about FlexFabric CNA Firmware and FCoE Drivers are available in the Virtual Connect FCoE Modules section
of the following SPOCK webpage:
http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_virtual_connect.html
(Requires HP Passport; if you do not have an HP Passport account, follow the instructions on the webpage).
These Virtual Connect Connectivity Stream documents provide a detailed table of the FlexFabric CNA firmware and FCoE
drivers that are supported by HP.
Note: For more information about HP Service Pack for ProLiant and Hotfixes, see
http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

Storage Array
SPOCK is the HP authoritative source of interoperability information for HP storage products. A particular configuration
is supported if and only if, it is listed on SPOCK.
HP Storage Arrays supported with Virtual Connect Dual-Hop configuration are listed in the Virtual Connect Connectivity
Stream documents available on the following SPOCK webpage:
http://h20272.www2.hp.com/Pages/spock2Html.aspx?htmlFile=hw_virtual_connect.html
For information about Non-HP Storage support, firmware versions and specific supported configuration information,
consult your storage equipment vendor.

4
Virtual Connect Limitations
 Only Dual-Hop FCoE is supported by HP at this time.

Figure 1: Dual-Hop FCoE Split FC & Ethernet at Top of Rack FCoE switch

LAN SAN

FCoE switch FCoE switch

FCoE C7000 with FCoE

Virtual Connect

C7000 with
Virtual Connect

 Only one FCoE network can be associated with any single set of uplink ports.

FCoE-capable Shared Uplink Set (SUS) can be used to allow concurrent Ethernet and FCoE traffic, but only one of the
networks in the SUS can be an FCoE Network.

Figure 2: Virtual Connect SUS configuration when using FCoE

Converged-SUS defined to pass

FCoE traffic and traditional VC Ethernet networks
Ethernet traffic

SUS-1 VLAN-30

VLAN-20
FCoE-1
VLAN-10

FCoE Lossless Network

LACP

5
 Virtual Connect requires the creation of an Active/Active Shared Uplink Set configuration for the FCoE traffic.

Figure 3: Active-Active FCoE Virtual Connect configuration

FCoE ToR FCoE ToR

Switch 1 Switch 2

Lossless Ethernet Network

Active Active
FCoE FCoE

SUS-1 SUS-2

VC Domain

FCoE ToR FCoE ToR

Switch 1 Switch 2

Lossless Ethernet Network

FCoE FCoE

SUS

VC Domain

 FCoE-capable Shared Uplink Sets must always contain ports from a single VC module in order to maintain the SAN-A/B
isolation.

Figure 4: FCoE-capable Shared Uplink Set configuration support

FCoE ToR FCoE ToR
Switch 1 Switch 2

FCoE FCoE

SUS-1 SUS-2

VC Domain

FCoE ToR FCoE ToR

Switch 1 Switch 2

FCoE FCoE

SUS-1 SUS-2

VC Domain

6
 FCoE-capable Shared Uplink Sets must always be connected to a single Top of Rack switch in order to maintain the
SAN-A/B isolation.

Figure 5: FCoE-capable Shared Uplink Set LACP configuration support

FCoE ToR FCoE ToR

Switch 1 Switch 2

Lossless Ethernet
Network
Active LACP LACP Active
FCoE FCoE

SUS-1 SUS-2

VC Domain

FCoE ToR FCoE ToR

Switch 1 Switch 2

Lossless
Ethernet
Network
Active Active
LACP LACP
FCoE FCoE

SUS-1 SUS-2
Ethernet forwarding through a
link aggregation can break
VC Domain SAN A/B separation

7
Dual-Hop FCoE with Nexus 5xxx Series ToR switch in FCF
mode scenario

In this scenario, the Nexus switches operate as Fiber Channel Forwarders (FCF). This is the default Cisco Nexus 5xxx
Series switches mode, it’s also called the fabric mode. In this mode, the switch provides standard Fibre Channel
switching capability and features.

Figure 6: Cisco Nexus 5xxx switches operating as FC Forwarder (FCF)

FC
Storage
Array

N_Port
Native FC
Lossless Ethernet network
Native FC

FCF F_Port F_Port FC

Storage
VN_Port VF_Port E_Port E_Port N_Port Array

Server Virtual Connect Nexus 5xxx Cisco

(ENode) (FIP Snooping mode) Series MDS
(Fabric Mode)

Dual-Hop FCoE

FCoE
FC

This cookbook provides only basic Cisco guidelines and configuration steps, for a more complete and detailed list of
Nexus FCoE configuration guidelines, refer to the Cisco NX-OS Fibre Channel over Ethernet Configuration Guides:

 Cisco Nexus 5000 Series NX-OS Fibre Channel over Ethernet Configuration Guide, Release 5.2(1)N1(1)
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/fcoe/521n11/b_5k_FCoE_Config_521N1
1.pdf
 Cisco Nexus 5500 Series NX-OS Fibre Channel over Ethernet Configuration Guide, Release 6.0(2)N1(1)
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/fcoe/602_N1_1/b_5500_FCoE_Config_6
02N11.pdf

8
Figure 7: Physical view

FC Storage Array

FC SAN A FC SAN B
Cisco MDS Series
SAN Switch

FC-FCoE
LAN

FC FC
Nexus 5xxx Series
ToR Switch using
Fabric Mode

FCoE FCoE
FCoE
HP Virtual Connect FlexFabric
or Flex-10/10D modules

VC Domain

Requirements
 Minimum NX-OS version 5.2(1)N1(3)
 FCoE requires the Nexus Storage Protocols Services license (FC_FEATURES_PKG)

Note: The license is an option which is activated when the Nexus switch is shipped from Cisco.

Guidelines
 Nexus switches must either bridge to native FC infrastructure or directly connect to FC/FCoE-based Storage devices.
Refer to the Cisco Nexus or Storage vendor interoperability Matrix.
 When bridged to a native FC infrastructure, it is mandatory to use Cisco MDS directors or fabric switches in order to
provide interoperability between fabrics.
 To increase the FCoE traffic identification and to better control the span of this traffic over the Ethernet network, it is
recommended to use different FCoE VLANs and VSANs numbers between the two fabrics.
 The FCoE VLAN should be dedicated to FCoE traffic (i.e. it should not carry IP traffic).
 The FCoE VLAN must not be configured as a native VLAN (the VLAN that carries untagged traffic on trunk ports, by
default VLAN 1).
 Interfaces connecting to VC must be configured as trunk ports and STP edge ports. (STP does not run on FCoE VLANs
between FCFs (VE_Ports) but does run on FCoE VLANs towards the host (VF_Ports)).

9
Nexus configuration when using a single VC to Nexus link
Details about the configuration:
 Interfaces eth1/5 are connected to the VC modules.
 Interfaces fc2/1 are directly connected to Cisco MDS 9148 switches.
 VLAN IDs 200 and 201 are used for the FCoE networks.
 VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-FCoE networks).
 The vfc interface (virtual Fibre Channel interface) binds to eth1/5.

Figure 8: Physical diagram

3PAR
Storage Array
WWN: WWN:
21:53:00:02:ac:00:15:9d 20:53:00:02:ac:00:15:9d

Native
SAN-A SAN-B Cisco MDS Ethernet
LAN 9148
Native
Fibre Channel
VLAN 1,10,20 VLAN 1,10,20
eth1/17 VSAN 200 eth1/17 VSAN 201
FCF Switch A FCF Switch B
Nexus-5010 Nexus-5010
eth1/5 fc2/1 eth1/5 fc2/1
vfc 2005 vfc 2005
FCoE VLAN 200 FCoE VLAN 201
VLAN 1,10,20 VLAN 1,10,20
FCoE / DCB
X3 X3
VC FlexFabric VC FlexFabric
Bay 1 Bay 2
Blade Server

FCoE NIC FCoE NIC

CNA CNA
Port 1 Port 2
WWN: WWN: FCoE/DCB
50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22
FC
HP BladeSystem c7000 LAN

10
Nexus switch-A configuration
 Upgrade the first Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)

 Enable FCoE on the switch (disabled by default):

o configure terminal
o feature fcoe

Note: The Nexus will require a reload.

 Associate a VLAN with a VSAN for FCoE traffic:

o vlan 200
o fcoe vsan 200

 Create a virtual Fibre Channel interface to carry the FCoE traffic through eth1/5:
o interface vfc 2005
o bind interface ethernet 1/5
o no shutdown

 Create the VLANs for the IP traffic:

o vlan 1,10,20

 Create a trunk on the interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):
o interface Ethernet1/5
o description FCoE uplink to FlexFabric
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20,200
o spanning-tree port type edge trunk

 Assign the vfc interface and the Fibre Channel port to the FCoE VSAN:
o vsan database
o vsan 200
o vsan 200 interface fc2/1
o vsan 200 interface vfc 2005

 Configure the interface connected to the datacenter LAN:

o interface eth 1/17
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20

 Configuration of the zone:

a. Create zones:
o zone name fcoe-zone vsan 200
o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port}
o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the Blade FlexFabric Adapter port 1}
b. Create zoneset:
o zoneset name zoneset1 vsan 200
o member fcoe-zone
c. Activate zoneset:
o zoneset activate name zoneset1 vsan 200

11
Nexus switch-B configuration
 Upgrade the second Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)

 Enable FCoE on the switch (disabled by default):

o configure terminal
o feature fcoe

Note: The Nexus will require a reload.

 Associate a VLAN with a VSAN for FCoE traffic:

o vlan 201
o fcoe vsan 201

 Create a virtual Fibre Channel interface to carry the FCoE traffic through eth1/5:
o interface vfc 2005
o bind interface ethernet 1/5
o no shutdown

 Create the VLANs for the IP traffic:

o vlan 1,10,20

 Create a trunk on the interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):
o interface Ethernet1/5
o description FCoE uplink to FlexFabric
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20,201
o spanning-tree port type edge trunk

 Assign the vfc interface and the Fibre Channel port to the FCoE VSAN:
o vsan database
o vsan 201
o vsan 201 interface fc2/1
o vsan 201 interface vfc 2005

 Configure the interface connected to the datacenter LAN:

o interface eth 1/17
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20

 Configuration of the zone:

a. Create zones:
o zone name fcoe-zone vsan 201
o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port}
o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the Blade FlexFabric Adapter port 2}
b. Create zoneset:
o zoneset name zoneset1 vsan 201
o member fcoe-zone
c. Activate zoneset:
o zoneset activate name zoneset1 vsan 201

12
Nexus configuration when using a port channel between VC and Nexus

Current NX-OS limitations

 All ports in the FCoE port channel must belong the same Unified Port Controller (UPC) ASIC; you cannot configure port
channels across UPCs.
 Nexus 5010/5020 switches can support a maximum of four ports per port-channel when carrying FCoE traffic.
 Nexus 5548/5596 switches can support a maximum of eight ports per port-channel when carrying FCoE traffic.

Note: These limitations will be removed in a future NX software release.

Details about the configuration:

 Interfaces eth1/5 and eth1/6 are connected to the VC modules.
 Interfaces fc2/1 are directly connected to Cisco MDS 9148 switches.
 VLAN IDs 200 and 201 are used for the FCoE networks.
 VLAN IDs 1, 10 and 20 are the standard Ethernet networks (non-FCoE networks).
 The vfc interface (virtual Fibre Channel interface) binds to the Port Channel 200 configured with eth1/5 and eth1/6.

Figure 9: Physical diagram

3PAR
Storage Array
WWN: WWN:
21:53:00:02:ac:00:15:9d 20:53:00:02:ac:00:15:9d

Native
SAN-A SAN-B Cisco MDS Ethernet
LAN 9148
Native
Fibre Channel
VLAN 1,10,20 VLAN 1,10,20
VSAN 200 eth1/17 VSAN 201
eth1/17
FCF Switch A FCF Switch B
Nexus-5010 Nexus-5010
eth1/5 – eth1/6 fc2/1 eth1/5 – eth1/6 fc2/1
vfc 2005 vfc 2005
FCoE VLAN 200 FCoE VLAN 201
po200 VLAN 1,10,20 VLAN 1,10,20 po200
FCoE / DCB
X3 - X4 X3 – X4
VC FlexFabric VC FlexFabric
Bay 1 Bay 2
Blade Server

FCoE NIC FCoE NIC

CNA CNA
Port 1 Port 2
WWN: WWN:
50:06:0b:00:00:c3:1a:20 50:06:0b:00:00:c3:1a:22
FCoE/DCB
HP BladeSystem c7000 FC
LAN

13
Identifying ports that belong to the same Unified Port Controller
It is necessary while configuring the FCoE port channel group to use interfaces that belong to the same UPC ASIC (e.g.
Eth1/5 and Eth1/6 because both interfaces are mapped to the same ASIC, UPC#1). In order to identify ports that belong
to same UPC, it is necessary to use the show hardware internal command.

 On Nexus 5010/5020 switches:

The Nexus 5010 and 5020 switches use Gatos Unified Port Controller ASIC (UPC Gen 1). Each UPC ASIC have 4 ports
mapped to front panel ports. This can be seen with the following command:

Nexus-5010# show hardware internal gatos all-ports

Gatos Port Info:
name |log|gat|mac|flag|adm|opr|c:m:s:l|ipt|fab|xgat|xpt|if_index|diag
-------+---+---+---+----+---+---+-------+---+---+----+---+--------+----
xgb1/4 |3 |0 |0 |b7 |dis|dn |0:0:0:f|0 |55 |0 |2 |1a003000|pass
xgb1/3 |2 |0 |1 |b7 |dis|dn |0:1:1:f|1 |54 |0 |0 |1a002000|pass
xgb1/1 |0 |0 |2 |b7 |dis|dn |1:2:2:f|2 |56 |0 |4 |1a000000|pass
xgb1/2 |1 |0 |3 |b7 |en |up |1:3:3:f|3 |57 |0 |6 |1a001000|pass
xgb1/8 |7 |1 |0 |b7 |en |up |0:0:0:f|0 |50 |1 |2 |1a007000|pass
xgb1/7 |6 |1 |1 |b7 |dis|dn |0:1:1:f|1 |51 |1 |0 |1a006000|pass
xgb1/5 |4 |1 |2 |b7 |en |up |1:2:2:f|2 |53 |1 |4 |1a004000|pass
xgb1/6 |5 |1 |3 |b7 |en |up |1:3:3:f|3 |52 |1 |6 |1a005000|pass
xgb1/19|18 |2 |0 |b7 |en |up |0:0:0:f|0 |45 |4 |0 |1a012000|pass
xgb1/11|10 |2 |1 |b7 |dis|dn |0:1:1:f|1 |44 |2 |0 |1a00a000|pass
xgb1/20|19 |2 |2 |b7 |en |up |1:2:2:f|2 |49 |4 |2 |1a013000|pass
xgb1/12|11 |2 |3 |b7 |dis|dn |1:3:3:f|3 |48 |2 |2 |1a00b000|pass
xgb1/18|17 |3 |0 |b7 |dis|dn |0:0:0:f|0 |41 |4 |6 |1a011000|pass
xgb1/10|9 |3 |1 |b7 |en |dn |0:1:1:f|1 |40 |2 |6 |1a009000|pass
xgb1/17|16 |3 |2 |b7 |en |up |1:2:2:f|2 |46 |4 |4 |1a010000|pass
xgb1/9 |8 |3 |3 |b7 |dis|dn |1:3:3:f|3 |47 |2 |4 |1a008000|pass
xgb2/2 |1 |4 |0 |b7 |dis|dn |0:0:0:f|0 |37 |5 |2 |1a081000|pass
xgb1/15|14 |4 |1 |b7 |dis|dn |0:1:1:f|1 |36 |3 |0 |1a00e000|pass
…

Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc.
Column 3- Indicates the UPC ASIC number.
Column 4- Indicates the UPC ASIC interface number. 0-3 are the 4 port available on the Gen-1 ASIC.

Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example,
Eth1/1 to Eth1/4 can be used in the same port channel because they are all mapped to UPC#0.

14
 Figure 10: Interface Eth1/5 cannot be part of a same Port Channel as Eth1/1 – Eth1/4

LACP

10G Ethernet
Interfaces
Eth1/1 Eth1/2 Eth1/3 Eth1/4 Eth1/5 Eth1/6 ...
xgb1/3 = Eth1/3

01 2 3 0 1 23

UPC #0 UPC #1

Only these interfaces

can be part of the same
FCoE Port Channel

 On Nexus 5548/5596 switches:

The Nexus 5548 and 5596 switches use Carmel Unified Port Controller ASIC (UPC Gen 2). Each UPC ASIC have 8 ports
mapped to front panel ports. This can be seen with the following command:

Nexus-5548# show hardware internal carmel all-ports

Carmel Port Info:
name |log|car|mac|flag|adm|opr|m:s:l|ipt|fab|xcar|xpt|if_index|diag|ucVer
-------+---+---+---+----+---+---+-----+---+---+----+---+--------+----+-----
xgb1/1 |0 |0 |0 -|b7 |dis|dn |0:0:f|0 |92 |0 |0 |1a000000|pass| 0.00
xgb1/2 |1 |0 |1 -|b7 |dis|dn |1:1:f|1 |88 |0 |0 |1a001000|pass| 0.00
xgb1/3 |2 |0 |2 -|b7 |dis|dn |2:2:f|2 |93 |0 |0 |1a002000|pass| 0.00
xgb1/4 |3 |0 |3 -|b7 |dis|dn |3:3:f|3 |89 |0 |0 |1a003000|pass| 0.00
xgb1/5 |4 |0 |4 -|b7 |dis|dn |4:4:f|4 |90 |0 |0 |1a004000|pass| 0.00
xgb1/6 |5 |0 |5 -|b7 |dis|dn |5:5:f|5 |94 |0 |0 |1a005000|pass| 0.00
xgb1/7 |6 |0 |6 -|b7 |dis|dn |6:6:f|6 |95 |0 |0 |1a006000|pass| 0.00
xgb1/8 |7 |0 |7 -|b7 |dis|dn |7:7:f|7 |91 |0 |0 |1a007000|pass| 0.00
xgb1/9 |8 |1 |0 -|b7 |dis|dn |0:0:f|0 |80 |0 |0 |1a008000|pass| 0.00
xgb1/10|9 |1 |1 p|b7 |en |up |1:1:f|1 |87 |0 |0 |1a009000|pass| 0.00
xgb1/11|10 |1 |2 -|b7 |dis|dn |2:2:f|2 |81 |0 |0 |1a00a000|pass| 0.00
xgb1/12|11 |1 |3 -|b7 |dis|dn |3:3:f|3 |86 |0 |0 |1a00b000|pass| 0.00
…

Note: Column 1- Indicates the physical ports e.g., Eth1/1, Eth1/3, Eth1/9, Eth1/17, etc.
Column 3- Indicates the UPC ASIC number.
Column 4- Indicates the UPC ASIC interface number. 0-7 are the 8 port available on the Gen-2 ASIC.

Note: Only ports identified as mapped to the same UPC ASIC can be member of the same port channel. For example,
Eth1/1 to Eth1/8 can be used in the same port channel because they are all mapped to UPC#0.

15
Nexus switch-A configuration
 Upgrade the first Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)

 Enable FCoE and LACP on the switch (disabled by default):

o configure terminal
o feature fcoe
o feature lacp

Note: The Nexus can require a reload.

 Associate a VLAN with a VSAN for FCoE traffic:

o vlan 200
o fcoe vsan 200

 Create a port channel with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect
Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010:
o interface ethernet 1/5
o channel-group 200 mode active
o lacp rate fast
o interface ethernet 1/6
o channel-group 200 mode active
o lacp rate fast

 Create the VLANs for the IP traffic:

o vlan 1,10,20

 Create a trunk on the port channel interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):
o interface port-channel 200
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20,200
o spanning-tree port type edge trunk

 Create a virtual Fibre Channel interface to carry the FCoE traffic through the port channel:
o interface vfc 2005
o bind interface port-channel 200
o no shutdown

 Assign the vfc interface and the Fibre Channel port to the FCoE VSAN:
o vsan database
o vsan 200
o vsan 200 interface fc2/1
o vsan 200 interface vfc 2005

 Configure the interface connected to the datacenter LAN:

o interface eth 1/17
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20

16
 Configuration of the zone:
a. Create zones:
o zone name fcoe-zone vsan 200
o member pwwn 21:53:00:02:ac:00:15:9d {This is the WWN of the first 3PAR controller port}
o member pwwn 50:06:0b:00:00:c3:1a:20 {This is the WWN of the Blade FlexFabric Adapter port 1}
b. Create zoneset:
o zoneset name zoneset1 vsan 200
o member fcoe-zone
c. Activate zoneset:
o zoneset activate name zoneset1 vsan 200

Nexus switch-B configuration

 Upgrade the first Nexus switch with minimum System version 5.2(1)N1(3) (enter: show version)

 Enable FCoE and LACP on the switch (disabled by default):

o configure terminal
o feature fcoe
o feature lacp

Note: The Nexus will require a reload.

 Associate a VLAN with a VSAN for FCoE traffic:

o vlan 201
o fcoe vsan 201

 Create a port channel with eth1/5 and eth1/6 with the same LACP Timer as defined by default in the Virtual Connect
Domain, eth1/5 and eth1/6 can be used because both are mapped to the same UPC (i.e. UPC#1) on Nexus 5010:
o interface ethernet 1/5
o channel-group 200 mode active
o lacp rate fast
o interface ethernet 1/6
o channel-group 200 mode active
o lacp rate fast

 Create the VLANs for the IP traffic:

o vlan 1,10,20

 Create a trunk on the port channel interface to pass the FCoE (VLAN 200) and Ethernet traffic (VLAN 1,10,20):
o interface port-channel 200
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20,201
o spanning-tree port type edge trunk

 Create a virtual Fibre Channel interface to carry the FCoE traffic through the port channel:
o interface vfc 2005
o bind interface port-channel 200
o no shutdown

17
 Assign the vfc interface and the Fibre Channel port to the FCoE VSAN:
o vsan database
o vsan 201
o vsan 201 interface fc2/1
o vsan 201 interface vfc 2005

 Configure the interface connected to the datacenter LAN:

o interface eth 1/17
o switchport mode trunk
o switchport trunk allowed vlan 1,10,20

 Configuration of the zone:

a. Create zones:
o zone name fcoe-zone vsan 201
o member pwwn 20:53:00:02:ac:00:15:9d {This is the WWN of the second 3PAR controller port}
o member pwwn 50:06:0b:00:00:c3:1a:22 {This is the WWN of the Blade FlexFabric Adapter port 2}
b. Create zoneset:
o zoneset name zoneset1 vsan 201
o member fcoe-zone
c. Activate zoneset:
o zoneset activate name zoneset1 vsan 201

18
Virtual Connect configuration
Virtual Connect requires the creation of an Active/Active configuration of two converged Shared Uplink Sets (SUS) which
contain one FCoE network and optionally associated non-FCoE network(s).

Figure 11: Virtual Connect Active-Active Converged Shared Uplink Sets scenario

Nexus 5xxx Series

ToR Switch using FCF Mode FCF Mode
Fabric Mode

802.1q 802.1q
Active 802.3ad Active 802.3ad Lossless
Ethernet
X1 X2 X1 X2 Network

VC Module bay 1 VC Module bay 2

Converged DCB-1 DCB-2

Shared Uplink Sets
VC Domain

FlexNIC FlexHBA FlexNIC FlexNIC FlexNIC FlexHBA FlexNIC FlexNIC

FlexFabric FlexFabric
Adapter Port 1 Adapter Port 2

vSwitch or NIC Teaming

Multi Pathing Blade Server

19
An alternative scenario can be created using dedicated Share Uplink Sets for the FCoE traffic combined with Shared
Uplink Sets for the standard Ethernet traffic since it is not mandatory to associate non-FCoE networks to a Converged
Shared Uplink Set network.

A typical scenario for this is when the Virtual Connect Domain cannot be modified and is already configured with Shared
Uplink Sets that are using either Non-FCoE capable VC uplink ports or Dual Homed LACP configurations (it is not
supported to connect a FCoE-capable Shared Uplink Set to two Top of Rack switches).

This scenario is using additional uplink ports however these extra links used by the FCoE traffic can provide a better
service level because the link is dedicated to the FCoE traffic and is not shared with any other types of network.

Figure 12: Alternative Virtual Connect Active-Active Converged Shared Uplink Sets scenario with Dedicated SUS for the
FCoE traffic.

Nexus 5xxx Series

FCF Mode vpc FCF Mode
ToR Switch using peer link
Fabric Mode

Active Dual Homed LACP

Active SUS configuration
Non-FCoE Non-FCoE
FCoE FCoE

VC Module bay 1 VC Module bay 2

Shared Uplink Set (Non-FCoE traffic) SUS-1 SUS-2

Converged Shared Uplink Set (FCoE traffic) SUS-FCoE-1 SUS-FCoE-2
Lossless Ethernet Network
VC Domain

FlexNIC FlexHBA FlexNIC FlexNIC FlexNIC FlexHBA FlexNIC FlexNIC

FlexFabric FlexFabric
Adapter Port 1 Adapter Port 2

vSwitch or NIC Teaming

Multi Pathing Blade Server

Converged Shared Uplink Sets (SUS) Details and Restrictions

 FCoE-capable Shared Uplink Sets (SUS) can contain both the FCoE network and non-FCoE networks.
 FCoE-capable SUS can support port aggregation (802.3ad).
 FCoE-capable SUS can support only one FCoE network.
 FCoE-capable SUS must always contain ports from a single VC module.

Note: For Multi Enclosure Stacking (MES) environments, all corresponding ports in remote enclosures will be included in
the same SUS. (e.g. selecting enc0:bay1:X1 means bay1:X1 in all remote enclosures is also included).

20
Defining two Converged Shared Uplink Sets

Create a SUS and name it “DCB-1”

 On the Virtual Connect Manager screen, click Define, Shared Uplink Set to create a SUS.

 Enter the Network Name of “DCB-1”.

 Select Add Port, then add one or more ports from Bay 1.

Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of
uplink to reach a good oversubscription ratio.

 For Connection Mode, use Auto.

 For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control
messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing in-
service upgrades on upstream switch firmware.
 For Associated FCoE Network, click on Add then enter:
o Network Name: FCoE-1.
o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the
upstream switch.
 For Associated Networks, click on Add then enter all Ethernet networks that are required.
o Select Smart Link for all defined networks.

21
Create a SUS and name it “DCB-2”
 Enter the Network Name of “DCB-2”.
 Select Add Port, then add one or more ports from Bay 2.

Note: In order to reduce the risk of congestion and increase the performance, make sure to use a suitable number of
uplink to reach a good oversubscription ratio.

 For Connection Mode, use Auto.

 For LACP Timer, leave the Domain Default, Short (1 sec). This setting controls the requested frequency of LACP control
messages on a LACP capable interface. Using the "long" setting can help prevent loss of LAGs while performing in-
service upgrades on upstream switch firmware.
 For Associated FCoE Network, click on Add then enter:
o Network Name: FCoE-2.
o VLAN ID: enter the VLAN ID used by the FCoE traffic, it should match with the FCoE VLAN ID used by the
upstream switch.
 For Associated Networks, click on Add then enter all Ethernet networks that are required.
o Select Smart Link for all defined networks.

Note: For the removal of ports from a Converged Shared Uplink Set, it is recommended to first shutdown or remove a
port from the vfc (interface vfc xx - no bind interface ethx/x) or channel group (interface ethx/x – no channel-group xx
mode active) on the FCF prior to removing the corresponding port from the Converged Shared Uplink Set in order to
allow the FCF to properly halt traffic on the link. Failure to do so may result in loss of storage access.

22
FCoE Server traffic configuration
Unifying storage traffic and network traffic on the same link can present some challenges and usually requires more
careful bandwidth management than standard Ethernet environments. One easy solution is to maintain the server
traffic to reasonable connection speed by using the VC Maximum connection speed option.
This option limits the server traffic by providing a maximum connection speed option for defined networks. This option
is recommended for high oversubscribed environment where the number of FCoE uplinks are limited which can increase
the risk of congestion and could badly impact the network performance.

Note: Maximum bandwidth optimization requires the Flex-10 and FlexFabric adapter firmware and drivers to be updated
to SPP version 2013.02.00 with Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01 compatibility, to take
advantage of this enhancement. This feature excludes support for the following adapters:

 HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter

 HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter
 HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter

To configure a maximum bandwidth limit from Server for the FCoE traffic:

 Go to the Ethernet Networks and select the first FCoE network and click Edit.

 Check the Advanced Network Settings option.

 Check Set Maximum Connection Speed.
 Select the best suitable speed for the maximum connection speed in order to meet your network design. Click Apply
and Apply again to save the new FCoE network configuration.

This value will limit the maximum port speed from Server for the FlexHBA associated with this FCoE network.

23
 Note: With 2 x 10Gb uplinks per VC modules connected to a Top of Rack switch using 50/50 bandwidth share and 16
servers using the FCoE network, it might be necessary to limit the maximum connection speed to get a reasonable
FCoE over-subscription:
o For a 2:1 oversubscription, use 1.3Gb
o For a 4:1 oversubscription, use 2.5Gb
o For a 8:1 oversubscription, use 5Gb
o For a 16:1 oversubscription, use 10Gb

 Repeat the same steps for the second FCoE network.

24
QoS Configuration
The default Virtual Connect configuration provides lossless service of the FCoE traffic. PFC (Priority Flow Control)
mechanisms are used to provide the FCoE lossless service and prevent congestion. It is therefore not necessary to
enable any particular QoS settings in the VC Domain.
Since Virtual Connect provides only best effort service for Ethernet traffic (i.e. flow control does not take place, in the
event of congestion, packets get dropped) Administrator can decide to configure QoS in the Virtual Connect Domain and
on the Nexus switches to improve the Ethernet traffic service level control.
The bandwidth allocation for FCoE-capable SUS uplinks always allocates only half the link bandwidth to FCoE traffic,
with the other half allocated to Ethernet traffic. With VC 4.01, this allocation is fixed and cannot be modified.
Therefore it is important to configure the Nexus switches with the same Virtual Connect Domain QoS settings as follow:

 FCoE lossless traffic class 3.

 50/50% of bandwidth sharing between regular and lossless traffic.
Conflicting QoS parameter settings between the Virtual Connect Domain and the Nexus environment can badly impact
the traffic and affect the performance.

Quality of Service configuration on the Nexus 5xxx series

QoS on Nexus 5010 and 5020
On the Nexus 5010 and 5020 it is not necessary to modify the QoS settings because by default once the FCoE feature is
configured (by entering “feature fcoe”) the default QoS template for shared links allocates the same shares and
priorities than Virtual Connect. The following two classes of service are automatically configured:

 class-fcoe is configured to be no-drop with a COS priority 3 and a MTU of 2158 with an ingress/egress bandwidth
allocation of 50%.
 class-default is configured to be drop with an MTU of 1500 with an ingress/egress bandwidth allocation of 50%.

25
To verify that the FCoE policy maps can be found in the running configuration, enter:

 show policy-map system

QoS on Nexus 5500 series

On the Nexus 5500 series, enabling the FCoE feature does not create automatically no-drop policies like on the Nexus
5000 series, so before enabling FCoE, it is necessary to configure the QoS policies manually by including class-fcoe in
each of the following policy types:

 Network-Qos
 Queuing
 Qos

The following is an example of a service policy that needs to be configured:

o class-map type qos class-fcoe
o class-map type queuing class-fcoe
o match qos-group 1
o class-map type queuing class-all-flood
o match qos-group 2

26
 o class-map type queuing class-ip-multicast
o match qos-group 2
o class-map type network-qos class-fcoe
o match qos-group 1
o class-map type network-qos class-all-flood
o match qos-group 2
o class-map type network-qos class-ip-multicast
o match qos-group 2
o system qos
o service-policy type qos input fcoe-default-in-policy
o service-policy type queuing input fcoe-default-in-policy
o service-policy type queuing output fcoe-default-out-policy
o service-policy type network-qos fcoe-default-nq-policy

To verify the FCoE system class is active on the interface connected to VC, enter:
 show queuing interface Ethernet 1/5

For more detailed description of the QoS configuration on Nexus switches, see the Cisco Nexus 5000 Series NX-OS Quality
of Service Configuration Guide at the following URL:
http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html

27
Quality of Service configuration on the Virtual Connect modules
It is not necessary to enable any particular QoS settings in the VC Domain to provide Quality of Service for the FCoE
traffic. The moment FCoE is configured, Virtual Connect provides lossless service for the FCoE traffic.
By default, the VC QoS configuration mode is Passthrough. In this mode, two classes of service are provided, one for
FCoE with lossless service and one for Ethernet without any form of flow control.
To improve the Ethernet traffic service level control and make VC to take a specific action on a specified classified traffic,
the VC QoS settings can be changed from Passthrough to Custom (with FCoE Lossless). In this mode, VC monitors the
Ethernet traffic for Layer 2 802.1p Priority bits, or Layer 3 DSCP or ToS markings and places packets on the pre-defined
egress queues. The Ethernet traffic is then prioritized based on the traffic classification.

Optional VCM QoS Configuration

 On the Virtual Connect Manager screen, click Configure then Quality of Service (QoS).

 Change the default Pass-through mode to Custom (with FCoE Lossless).

28
 In this mode, VC supports up to 8 configurable traffic classes:
o 1 predefined system class for Best Effort.
o 1 predefined system class for FCoE Lossless.
o 6 user defined classes.

Note: The default 802.1p priority for the Lossless FCoE traffic class is 3.

Note: The FCoE Lossless traffic share is based on the VC server profile configuration. The MAX Share is based on the
FCoE Fabric configuration.

 For each user defined classes you want to use, check the Enabled box and enter the appropriate Share/Max share and
802.1p priority.

 Then click Apply.

29
Connecting Servers to the FCoE networks

To connect Blade servers to FCoE networks that have been configured, go to the Server Profiles interface.
 Select a profile and click Edit.

 Select the first FCoE HBA Connection and select the first FCoE network FCoE-1.

The server administrator can select the FCoE network like any other storage network when configuring profiles.

30
 Select the second FCoE HBA Connection and select the second FCoE network FCoE-2.

 The two Active/Active FCoE networks are now assigned to the server FCoE HBA.

To configure a Minimum Allocated Port Speed for the FCoE traffic:

 Select Custom from the Port Speed Type drop down menu of the first FCoE port.

 Then select the port speed, this speed is the minimum guaranteed bandwidth.

Maximum Connection
Speed that had been
defined for FCoE-1

Note: Make sure to select the best suitable Minimum Port Speed connection in order to meet your network design.

 Click Ok.

Note: The Minimum Allocated Port Speed feature is an additional solution to overcome the challenges of network
convergence discussed before. This Virtual Connect option guarantees that the FCoE traffic from server will get its
minimum allocated bandwidth even if other FlexNICs on the same port generate too much traffic. Furthermore, if a
congestion event occurs, the bandwidth is allocated based on the ratio of the Minimum setting. This option can really
improve the Virtual connect bandwidth utilization and network performance.

31
 Note: Minimum and maximum bandwidth optimization requires the Flex-10 and FlexFabric adapter firmware and
drivers to be updated to SPP version 2013.02.00 with Hotfix (10.70) - Linux Drivers and CNA Firmware for VC 4.01
compatibility, to take advantage of this enhancement. This feature excludes support for the following adapters:
o HP NC551i Dual Port FlexFabric 10Gb Converged Network Adapter
o HP NC551m Dual Port FlexFabric 10Gb Converged Network Adapter
o HP NC550m 10Gb 2-port PCIe x8 Flex-10 Ethernet Adapter

 Repeat the same steps for the second FCoE port.

 Click Apply to activate the profile configuration.

 The Allocated Port Speed should now be displayed with the Min and Max that have been defined.

 The next step is to configure under the Server Operating System the NIC teaming and the FC multi-pathing for
failover.

32
Nexus Troubleshooting commands
For more detailed information about Nexus FCoE troubleshooting guidance, see
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/troubleshooting/guide/n5K_ts_fcoe.html
Disclaimer: we used the Cisco Nexus 5000 FCoE Troubleshooting Guide to create this section.

Useful Show commands

 show flogi database

o This command displays all logins (server nodes and storage devices) which have logged into the Fabric using
FC and FCoE. Each FCoE Adapter port logs into the Fabric using the same virtual FC interface. The vfc are
listed along with the VC assigned WWN addresses, the VSAN numbers and the FCIDs assigned by the Fabric to
every single server.

 show fcoe database

Server #1
Server #2
Server #3

o This command is the same as the previous one but only displays the FCoE logins.
o It provides also the VC assigned MAC address of the FCoE adapter.
o Interfaces are listed with their FCIDs assigned by the Fabric and their PWWNs and MAC addresses assigned by
Virtual Connect.

 show fcoe

o This command shows if FCoE is enabled.

33
 o It displays the FCF-MAC and the FC-MAP of the Nexus device. This command is useful because every FCoE
frame sent by servers always use this FCF-MAC address as their Destination MAC Address.

 show fcdomain

 show vsan 200 membership

34
 show policy-map

o 2 traffic classes are defined, one for the FCoe traffic and one for Ethernet.

 show class-map class-fcoe

o The FCoE traffic class is defined with priority 3.

Note: COS 3 is the default Industry standard value for the FCoE Lossless traffic class (IEEE 802.1p). The same priority
is used by the Virtual Connect Domain.

35
 show queuing interface ethernet 1/5

50% of the link

bandwidth is
reserved to the
FCoE traffic

qos-group 1
defines the
lossless type of
service (FCoE).

o This commands shows the ETS (Enhanced Transmission Selection - IEEE 802.1Qaz) configuration of an
interface.
o It shows two traffic classes, one for the FCoE traffic (no-drop, MTU 2240) and one for the Ethernet traffic
(drop, MTU 1500).
o The bandwidth allocation for each traffic class is 50% of the link bandwidth.

 show logging last 20

o This command displays the last 20 system log messages of the device.
o Always very helpful, the Nexus logs can provide useful messages to help the troubleshooting.

 show logging | grep vfc

o This command is useful to check the status of the virtual interface used for the FCoE traffic.
o The vfc can be up, down, administratively down, etc.
o A “down but waiting for flogi” can mean that the setup is ok but no blade server has been turned on yet.

36
Per-Priority Flow Control commands
Make sure PFC (Per-priority Flow Control - 802.1Qbb) is enabled on the interface connected to VC. Flow Control must be
disabled. PFC ensures the lossless behavior of the FCoE traffic.

 Check the status of Flow Control; it must be turned off on interfaces connected to Virtual Connect.
o show interface ethernet 1/5 flowcontrol

 Check the status of the PFC. It must be turned on on interfaces connected to Virtual Connect.
o show interface ethernet 1/5 priority-flow-control

 If the command shows “Mode” as Off or On, it should be changed to Auto.

To force PFC mode Auto on an interface, enter:
 interface ethernet 1/5
 priority-flow-control mode auto

 If the command shows “Mode” as Auto and “Oper” as Off or if the command returns
nothing, you should check the setup of the Shared Uplink Set in VCM by making sure that an
associated FCoE Network has been properly defined.

37
Issues with FCoE performance
If FCoE throughput on servers is very low, this can be due to pause frames. It is necessary to check if the switch is
sending pause frames or if it is getting paused.

Possible Cause:
 If the egress FC port is congested, then the switch will send PFC frames to the Virtual Connect uplink port. The PFC
frames are sent to reduce its FCoE rate and avoid a drop.
 If the server is slow or congested, then the server will send PFC frames to the Virtual Connect downlink port. As a
result Virtual Connect will eventually send PFC Frames to the Nexus interface.

To verify this situation, it is necessary to check the Nexus interface that is connected to the Virtual Connect modules.

When slow FCoE performance is not due to Pause Frames

 For a few iterations using the "show interface ethx/x priority-flow-control" command, the PFC frame count (RX/TX)
is not incrementing:

 For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for both RX
and TX as Inactive:

38
Solution:
The slow FCoE performance does not seem to be due to network congestion. Check the Nexus interface to find
information about possible link issues:

 Use the show interface command to verify that the Nexus interface connected to VC is not showing errors, drops,
discards, etc.
 Use the show interface command to verify that the vfc interface is not showing errors or discards.
 Use the show queuing interface to verify the number of discards.

When slow FCoE performance is due to Pause frames transmitted by the Nexus switch.

Figure 13: Nexus sending PFC Pause frames due to FC congestion.

FC Target

LAN
Congestion
Point

PPP
Nexus interface connected
to Virtual Connect that needs
to be monitored

39
When the Nexus switch is sending Pause frames to VC, the Nexus interface that is connected to the Virtual Connect
module is showing the following:

 For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for TX as
Active:

 For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause TxPPP counter is
incrementing:

Solution:
Identify the source of the congestion and try to resolve it by increasing the FC bandwidth (or by adding additional Nexus
FC ports to connect to the SAN environment).

40
When slow FCoE performance is due to Pause frames transmitted by VC/Servers.

Figure 14: Virtual Connect sending PFC Pause frames due to congestion.

FC Target

LAN

Congestion PPP
Point
Nexus interface connected
to Virtual Connect that needs
to be monitored

PPP
Congestion
Point

When the Virtual Connect uplink is sending Pause frames to the Nexus, the Nexus interface that is connected to the
Virtual Connect module is showing the following:.

 For a few iterations using the "show queuing interface ethx/x" command, the pause status are detected for Rx as
Active as shown below:

41
 For a few iterations using the "show interface ethx/x priority-flow-control" command, the pause RxPPP counter is
incrementing:

Solution:
Identify the source of the congestion and try to resolve it with the following action:
1. Increase the FCoE connection speed (Min and Max) of every servers.
2. Increase the number of VC FCoE Uplink ports connected to the Nexus.

Note: If congestion is expected, then Pause is expected for FCoE traffic.

Note: PFC Pause frames are currently not available in the Virtual Connect statistics. Only the standard Ethernet Pause
frames are displayed.

42
Interface Status commands
To check the status of the port connected to VC, you can enter:

 show interface status err-disabled

VFC is usually down due to FIP solicitation failure and possible reasons are:
o Fabric is not available for flogi.
o Zoning or Storage presentation are misconfigured.

 show interface vfc2005

 show interface vfc2005 brief

43
FCoE Frames Analysis
For network troubleshooting, capturing frames with a protocol analyzer can be very instructive but it is necessary to
understand the way FCoE works.

FCoE traffic use a different MAC address than the MAC address of the FCoE Adapter, the physical MAC address of the
FCoE adapter (i.e. the MAC displayed in the Virtual Connect interface) is only used during FIP negotiation (FCoE
Initialization Protocol).

Figure 15: MAC address of the FCoE adapter found in the Server profile interface.

During the FIP negotiation, the FCF assigned a Fabric Provided MAC address (FPMA) to the server. This address is used for
FCoE forwarding and is the one Wireshark will detect during FCoE network captures.

The FPMA is built by appending the FCID (assigned by the Fabric to the server) to the Nexus FC-MAP.

Figure 16: The way the FPMA is built.

24 Bits 24 Bits
FC-MAPs FC-ID
0E-FC-00 to 0E-FC-FF E5.01.00

FC-MAP FC_ID
0E-FC-00 E5.01.00
0E-FC-00-E5-01-00

FPMA
48 bits

44
FPMA addresses assigned to servers cannot be found from the Virtual Connect interface, it is necessary to launch the
Nexus console and use the following commands:
 show fcoe
 show fcoe database

Figure 17: How to find Fabric Provided MAC Addresses.

FPMA of server 1 : Oe:fc:00:e5:01:00

FPMA of server 2 : Oe:fc:00:e5:03:00

FPMA of server 3 : Oe:fc:00:e5:02:00

FCoE Frames sent by servers:

 Destination MAC: the destination of FCoE frames sent by servers is always the MAC address of the Fibre Chanel
Forwarder (i.e. the FCF-MAC address of the Nexus switch).
 Source MAC: the source of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server by the FCF
during the FIP negotiation.

FCoE Frames received by servers:

 Destination MAC: the destination of an FCoE frame is the Fabric Provided MAC Address (FPMA) assigned to the server
by the FCF during the FIP negotiation.
 Source MAC: the source of FCoE frames sent by the FCF to a server is always the MAC address of the Fibre Chanel
Forwarder (i.e. the FCF-MAC address of the Nexus switch).

45
Figure 18: Sample of an FCoE frame sent by a server

FPMA (FC-MAP+FCID)

FCID of the FCoE Server

FCID of the Storage Target

46
Support and Other Resources
Contacting HP
Before you contact HP
Be sure to have the following information available before you call contact HP:

 Technical support registration number (if applicable)

 Product serial number
 Product model name and number
 Product identification number
 Applicable error message
 Add-on boards or hardware
 Third-party hardware or software
 Operating system type and revision level

HP contact information
For help with HP Virtual Connect, see the HP Virtual Connect webpage: http://ww.hp.com/go/virtualconnect
For the name of the nearest HP authorized reseller:
See the Contact HP worldwide (in English) webpage:
http://www.hp.com/country/us/en/wwcontact.html
For HP technical support:
In the United States, for contact options see the Contact HP United States webpage:
http://welcome.hp.com/country/us/en/contact_us.html
To contact HP by phone:

 Call 1-800-HP-INVENT (1-800-474-6836). This service is available 24 hours a day, 7days a week. For continuous
quality improvement, calls may be recorded or monitored.

 If you have purchased a Care Pack (service upgrade), call 1-800-633-3600. For more information about Care Packs,
refer to the HP website:
http://www.hp.com/hps

 In other locations, see the Contact HP worldwide (in English) webpage:

http://welcome.hp.com/country/us/en/wwcontact.html

Subscription service
HP recommends that you register your product at the Subscriber's Choice for Business website:
http://www.hp.com/country/us/en/contact_us.html
After registering, you will receive email notification of product enhancements, new driver versions, firmware updates,
and other product resources.

47
Related documentation
HP Virtual Connect Manager 4.01 Release Notes
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03801912/c03801912.pdf

HP Virtual Connect for c-Class BladeSystem Version 4.01 User Guide

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03791917/c03791917.pdf

HP Virtual Connect Version 4.01 CLI User Guide

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03790895/c03790895.pdf

HP Virtual Connect for c-Class BladeSystem Setup and Installation Guide Version 4.01 and later
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03801914/c03801914.pdf

HP Virtual Connect FlexFabric Cookbook

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02616817/c02616817.pdf

HP Virtual Connect Fibre Channel Networking Scenarios Cookbook

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01702940/c01702940.pdf

HP Virtual Connect with iSCSI Cookbook

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02533991/c02533991.pdf

HP Virtual Connect Multi-Enclosure Stacking Reference Guide

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02102153/c02102153.pdf

Converged Networks and Fibre Channel over Ethernet Technical Whitepaper

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c03440617/c03440617.pdf

HP welcomes your feedback. To make comments and suggestions about product documentation, send a message to
docsfeedback@hp.com. Include the document title and manufacturing part number. All submissions become the
property of HP.

Get connected
hp.com/go/getconnected
Current HP driver, support, and security alerts
delivered directly to your desktop

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only
warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein
should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omiss
omissions
ions contained herein.

Trademark acknowledgments, if needed.

48
4AA4-xxxxENW,
c03808925, Created
Created
JuneMonth
2013 2013