BRKDCN 3346

#CLUS
End-to-End QoS
Implementation and
Operation with Nexus
Nemanja Kamenica
Technical Marketing Engineer
BRKDCN-3346
#CLUS
Session Objectives
• Provide a refresh of QoS Basics
• Understand the basic switch
architecture for the Nexus switch
family
• Provide a detailed understanding of
QoS on Nexus platforms
• Learn how to configure QOS on Nexus
devices through real-world
configuration examples
#CLUS BRKDCN-3346 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Non-Objectives
• Data Centre QoS Methodology
• Nexus hardware architecture deep-dive
• Application Centric Infrastructure (ACI) QOS
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKDCN-3346

by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Agenda
• Introduction
• QoS Basics
• QoS Implementation on Nexus
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Real World Configuration Examples
• Conclusion
Congestion Happens Everyday!
Why QoS in the Data Centre?
Assign Manage Maximise
Colour to Traffic Congestion Throughput
Maximise Throughput and Manage Congestion!
Can Traffic Control help …
… or confuse
… or hurt
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
The QoS Toolset 25nd Anniversary
Queuing and
Classification Policing Marking Shaping
Scheduling
Identify and Discard Mark Traffic Prioritise, Control

Split Traffic Misbehaving According to Protect and Bursts and
into Different Traffic to Maintain Behaviour Isolate Traffic Conform
Classes Network Integrity and Business Based on Traffic
Policies Markings
Traffic Management Tools
• Classification
• Traffic Categorisation based on
traffic attributes
• Marking
• Assigning different/new attribute
(priority) to traffic
• Policing
• Limit misbehaving flows
Classification and Marking – Two sides of a coin
• Identify traffic lync
mgmt p2p
Video
• DSCP Unclassified: VoIP
HTTP
email
App1
• IP PREC
Uncontrolled Bandwidth
vmotion
backup
• CoS
• ACLs
• Re-map Traffic
VoIP/Lync
• Like to Like (i.e. CoS to CoS)
Classified: Email/HTTP
• Like to Unlike (i.e. DSCP to COS)
Controlled Bandwidth SAP
• Needs mapping tables
Backup
• Also called Mutation
Policing – Limit Misbehaving Traffic
• Single rate Two Color Policer Tokens
• Conform Action (permit)

• Exceed Action (drop)
Burst
Limit
• Two rate Three Color Policer

• Conform Action (permit) Policer
• Exceed Action (markdown)
Conforming
• Violate Action (drop) Traffic
Excess Traffic
Congestion Management Tools
• Buffering
• Storing packets in memory
• Queuing
• Buffering packets according to traffic class
• Scheduling
• Order of transmission of buffered packets
• Shaping
• Smooth burst traffic
Buffering – Why do we need it?
• Many to One Conversations
• Client to Server
• Server to Storage
• Aggregation Points
• Speed Mismatch
• Client to WAN to Server
10 GE 10 GE 1 GE
1 GE
1 GE
1 GE
4 Class Queuing Model
• Matches most Service-Provider Class CoS Queues
offerings
Priority 5-7 PQ
• Ready for No-Drop traffic like FCoE
No-Drop 3 Q2
• One Class left to place traffic above
Better or
or below Best-Effort traffic priority
Worse then 1,2,4 Q1
• Special Application which is drop Best-Effort
sensitive (above Best-Effort -
Best-Effort 0 Default-Q
Critical)
• Non-Critical Bandwidth intensive
application (below Best-Effort -
Scavenger)
8 Class Queuing Model
• Matches often a Campus QoS concept
Class DSCP Queues
• DSCP to CoS derivation does NOT Priority CS6 (CS7)
apply anymore PQ
• (Topmost 3-Bit mapping from DSCP Platinum EF
to CoS) Gold AF41 Q7
• No-Drop still with CoS3 (DSCP 24-30 Silver CS4 Q6
are “unusable”) No-Drop CoS3 Q5
• Valid but most complex Classification Bronze AF21 Q4
to Marking implementation as per Management CS2 Q3
regards to No-Drop
Scavenger AF11 Q2
Bulk Data CS1 Q1
Best-Effort 0 Default-Q
Head of Line Blocking
What is the Problem?
Ingress Module Egress Module

buffer
Crossbar
Ingress Module Fabric Egress Module
Virtual Output Queues
Avoid Head of Line Blocking
Ingress
VOQ
Crossbar
Virtual Output Queues
Avoid Head of Line Blocking
Ingress
VOQ

buffer
Crossbar
Buffering on Nexus Models compared
VOQ
Output Queue Buffer
Virtual Output Queue
Ingress Egress Ingress Egress

Shared no-drop/
drop
N ports N ports N ports N ports
Input buffer for every egress port Shared buffer for N egress ports
NxN buffer size N buffer size
Scheduling – Who goes first?
• Defines Order of transmission
• The Priority-Queue always
serviced first
• Normal Queues serviced only
after Priority Queue empty
• Different Scheduling algorithms
for normal queues
Common Scheduling Algorithms
• Round Robin (RR) • Deficit Weighted Round Robin

• Simple and Easy to implement • Variable sized packets
• Starvation-free • Uses a deficit counter
• Weighted Round Robin (WRR) • Shaped Round Robin

• Serves n packets per non-empty • More even distributed ordering
queue • Weighted interleaving of flows
• Assumes a mean packet size
Congestion Avoidance Tools
• Tail Drop (TD)
• Drop packets at tail of the queue
• Single threshold per queue
• Weighted Random Early Drop (WRED)

• One or more thresholds per queue
• Threshold associated with DSCP or COS
Putting it all together! COS Queue
1 2 5, 6, 7 PQ1
Packet is placed in the Egress Priority Queue
Queue according to CoS/DSCP always served first 3, 4 Q3
value.
2 Q2
Egress Port
Normal Queue 0, 1 Q-
4
Default
Normal Queues have
Drop Threshold 4 Threshold and drop
PQ1 packet accordingly
Drop Threshold 3
Drop Threshold 2
Q2
Scheduler
Drop Threshold 1
3
DWRR
DSCP [24]
data L3
Normal Queues
L2
CoS [3] Q3 served after PQ is
empty according to
Scheduler
Q-Default
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
QoS
Implementation on
Nexus
Nexus uses Modular QOS CLI (MQC)
3 Block Construct
Class-Map
What Traffic do we care about? Policy-Map

• DSCP
• CoS
What actions do I take on the
• IPPREC
classes?
• ACLs
• Policing
Service-Policy
• Marking
• Scheduling
• Queueing Where do I apply this policy?
• System Wide
• VLAN
• Interface
• Port-channels
Three Different Types
Class-map Policy-map Service-policy

Queuing QoS
QoS QoS Queuing
Queuing • Buffering •Interfaces
• Marking •Interfaces
• CoS • CoS • Queuing •Vlans
• DSCP • Policing •Port-
• DSCP • Scheduli •Port-
• PREC • Mutation channels
ng channel
•System-qos
• ACLs •System-qos
Network-QoS Network-QoS
• CoS Network-QoS
• Congestion-Control
• Protocol (FCoE) • System-qos
• Pause / MTU per VL
Network-QoS Policy
• Define global queuing and scheduling parameters for all interfaces in switch
• Identify drop/no-drop classes, MTU and WRED/TD, etc.
• One network-QoS policy per system, applies to all ports
• Assumption is network-QoS policy defined/applied consistently network-wide
Network QoS policies should be applied consistently

on all switches network wide
Switch 1 Switch 2 Switch 3
Ingress Ingress Ingress

Module Module Module
Ingress Egress Ingress Egress Ingress Egress
Module Fabric Module Module Fabric Module Module Fabric
Module
Ingress Ingress Ingress
Module Module Module
System based Policy attachment
• System based QoS Policy gets
globally applied to all interfaces Routed Port
and VLAN Access Port
VLAN
• System based QoS Policy is 802.1q Trunk
QoS
configured in System QoS Routed Port Polcies
VLAN
Access Port
802.1q Trunk
Service Policy
applied to the
VLAN Database
Nexus(config)# system qos

Nexus(config-sys-qos)# service-policy input myPolicy
VLAN based QoS Policy attachment
• VLAN based QoS Policy is Routed Port
configured in VLAN Database Access Port

VLAN
QoS
Polcies
802.1q Trunk
• No SVI (aka L3 VLAN Interface) Routed Port
VLAN
required Access Port
802.1q Trunk
Service Policy
applied to the
VLAN Database
Nexus(config)# vlan configuration <vlan-id>

Nexus(config-vlan)# service-policy input myPolicy
Interface based QoS Policy attachment
• Interface based QoS Policy Service Policy
policy
Routed Port
takes precedence over VLAN applied to the

Switchport
switchport…
Access Port QoS
Policies
• Can also be attached to port- 802.1q Trunk
channel and applies to all

Routed Port
… or the
member-ports Routed Port Port-channel
• No Egress QoS policies on L2 … or a port-
ports! channel
Nexus(config)# interface ethernet 1/1

Nexus(config-if)# service-policy input myPolicy
Interface based Queuing Policy attachment
• Interface based QoS Policy Service Policy
policy
Routed Port
takes precedence over VLAN applied to the

Switchport
switchport…
Access Port QoS
Policies
• Interface based QoS Policy is 802.1q Trunk
configured under the

Routed Port
… or the
respective Interface Routed Port Port-Channel
• Queuing Policy can be … or a port-
attached to port-channel also channel
Nexus(config)# interface ethernet 1/1

Nexus(config-if)# service-policy input myPolicy
New QoS Capabilities
• Priority Flow Control
(802.1Qbb)
• Enables Lossless Ethernet using
per traffic class pause
• During congestion, no-drop
priority is paused
• No effect on other priority values
DC QoS Capabilities
• DCBXP (802.1Qaz)
• LLDP with new TLV Values
• Negotiates capabilities (like PFC) with
other devices
• ECN (Explicit Congestion Notification)

ECN ECN Behavior
• Congestion Notification without dropping
packets 0x00 Non ECN Capable
• Uses two LSB bits in DiffServ field

0x10 ECN Capable Transport (0)
IP header 0x01 ECN Capable Transport (1)
0x11 Congestion Encountered
Data Centre Converged Infrastructure
• Simplification of the
infrastructure by using Ethernet
for data and storage traffic
• FCoE
• Replaces Fibre Channel stack
with Ethernet
• RoCE
• RoCE extends RDMA capabilities
over Ethernet
RoCE vs RoCEv2 (non-drop) FC/FCoE
• Requirement for FCoE and
RoCEv1: FCoE RoCE v1 RoCE v2
• PFC Applications Applications Applications
• ETS FCP RDMA API RDMA API
Requirement for RoCEv2

FC Transport IB Transport IB Transport
•
FCOE IB Network UDP/IP
• PFC
Ethernet Ethernet Ethernet
• ETS
• ECN (optional)
To Trust or Not To Trust?
• Data Centre architecture
provides a new set of trust
boundaries
• Virtual Switch extends the trust
boundary into the Hypervisor
• Nexus Switches always trust
CoS and DSCP
Overlay QOS
Overlay QoS
MPLS network EXP COS DSCP IP pres
• Mapping between IP priorities to EXP
on PE router 0 0 0 0
1 1 8 1
• Classification is done biased on COS,
DSCP, IP precedence or ACL 2 2 16 2
• DiffServ Tunneling mode provides 3 3 24 3

different QOS behavior in provider 4 4 32 4
network
5 5 40 5
• Uniform mode delivers overlay priority
• Pipe mode extends underlay priority 6 6 48 6
7 7 56 7
Overlay QOS
MPLS – Default Mode
EXP 3 EXP 3
EXP 3 EXP 3 EXP 3
DSCP 24 DSCP 24 DSCP 24 DSCP 24 DSCP 24
CE1 PE1 P1 P2 PE2 CE2

MPLS Core
Overlay QOS
MPLS – Uniform Mode
EXP 3 EXP 2
EXP 3 EXP 3 EXP 2

MPLS Core
Overlay QOS
MPLS – Pipe Mode
EXP 3 EXP 5
EXP 3 EXP 3 EXP 5

MPLS Core
Overlay QoS
VXLAN EVPN – VXLAN Encapsulation
• Ingress L3 packet, original priority is mapped to outer header priority
• Ingress L2 frame, COS value will be mapped to outer priority
• VLAN header is not preserved in VXLAN tunnel COS DSCP
0 0
Original L3 Packet Original L2 Frame 1 8
COS 3 DSCP 26 COS 3 N/A 2 16
3 26
DSCP 26 DSCP 26 DSCP 26 N/A 4 32
VXLAN Encap. Packet 5 46

VXLAN Encap. Packet 6 48
7 56
Overlay QoS
VXLAN EVPN – VXLAN Decapsulation Uniform Mode
DSCP 26 DSCP 26
• DSCP value is derived based on a
priority mode for L3 traffic:
• Uniform mode: delivers overlay priority
copying outer header to decapsulated COS 0 DSCP 26
frame
• Pipe mode: extends original priority Pipe Mode
copying inner header to decapsulated DSCP 26 DSCP 26
frame
• Marking can be configure on the egress
VTEP mark decapsulated traffic with
COS 0 DSCP 26
priority (COS, DSCP)
Overlay QoS
VXLAN – Uniform Mode
Spine Spine
DSCP 26 DSCP 26 VXLAN EVPN DSCP 26 DSCP 26
VTEP VTEP VTEP VTEP VTEP
COS 3 DSCP 26 COS 0 DSCP 26

http
Baremetal Baremetal Baremetal
Host A Host C Host B

192.168.10.11 192.168.11.20 192.168.10.15
Overlay QoS
VXLAN – Pipe Mode
Spine Spine
DSCP 26 DSCP 26 VXLAN EVPN DSCP 26 DSCP 26
VTEP VTEP VTEP VTEP VTEP
COS 3 DSCP 26 COS 0 DSCP 26

http
Baremetal Baremetal Baremetal
Host A Host C Host B

192.168.10.11 192.168.11.20 192.168.10.15
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
Nexus 9000 QoS
Nexus 9000 Overview
• Modular and Fixed chassis
• Optimized for high density 10G/25G/40G/100G
• Standalone and ACI Mode
• Built with Cisco Silicon
• Advanced QoS capabilities
Nexus 9000 - Cloud Scale
Slice 0
LSE 900G
• 1.8T chip – 2 slices of 9 x 100G each Slice Interconnect
• X9700-EX modular linecards; 9300-EX TORs Slice 1 Slice 0
900G 1.8T
LS1800FX
• 1.8T chip – 1 slice of 18 x 100G with MACSEC LSE – 18 x 100G
• X9700-FX modular linecards; 9300-FX TORs
LS1800FX – 18 x 100G
S6400
Slice 0 Slice 1
• 6.4T chip – 4 slices of 16 x 100G each 1.6T 1.6T
Slice 0
• E2-series fabric modules; 9364C TOR
1.8T
Slice Interconnect
LS3600FX2 Slice Interconnect
• 3.6T chip – 2 slices of 18 x 100G with MACSEC + Slice 2 Slice 3
CloudSec 1.6T 1.6T Slice 1
1.8T
• 9300-FX2 TORs
S6400 – 64 x 100G
LS3600FX2 – 36 x 100G
What Is a “Slice”?
Slice
Ingress Slice 1 Interconnect
• Self-contained forwarding
Egress Slice 1
complex controlling subset of
ports on single ASIC
• Separated into Ingress and Ingress Slice 2
Egress functions Egress Slice 2
• Ingress of each slice connected

to egress of all slices
Ingress Slice n
• Slice interconnect provides non- Egress Slice n
blocking any-to-any
interconnection between slices
Cisco Nexus 9000 QoS Features
• Traffic classification
• DSCP, CoS, IP Precedence and ACL
• Packet marking
• DSCP, CoS, and ECN
• Strict Priority Queuing and DWRR
• Ingress and egress policing
• Tail Drop and WRED with ECN
• Shared buffer capability
• Egress Queuing
Buffering
• Cloud Scale platforms implement shared-memory egress buffered architecture
• Each ASIC slice has dedicated buffer – only ports on that slice can use that buffer
• Dynamic Buffer Protection adjusts max thresholds based on class and buffer occupancy
• Intelligent buffer options maximise buffer efficiency
Slice 0 Slice 0 Slice 1 Slice 0

18.7MB 10.2MB 10.2MB 20MB
Slice 0
Slice Interconnect Slice Interconnect Slice Interconnect
40.8MB
Slice 1 Slice 2 Slice 3 Slice 1
18.7MB 10.2MB 10.2MB 20MB
LSE LS1800FX S6400 LS3600FX2

18.7MB/slice 40.8MB/slice 10.2MB/slice 20MB/slice
(37.4MB total) (40.8MB total) (40.8MB total) (40MB total)
Queuing and Scheduling
50/50 DWRR
UC0 MC0 UC1 MC1 UC2 MC2 UC3 MC3 UC4 MC4 UC5 MC5 UC6 MC6 UC7 MC7
CPU Class 0 Class 1 Class 2 Class 3 Class 4 Class 5 Class 6 Class 7 SPAN
Strict Configurable Weights / Priority Best

Priority Effort
Egress
Port
Final Winner
• 8 user classes and 16 queues per output port (8 unicast, 8 multicast)
• QOS-group drives class; egress queuing policy defines class priority and weights
• Dedicated classes for CPU traffic and SPAN traffic
Intelligent Buffering
Innovative Buffer Management for Cloud Scale switches
• Dynamic Buffer Protection (DBP) – Controls buffer allocation for
congested queues in shared-memory architecture
• Approximate Fair Drop (AFD) – Maintains buffer headroom per
queue to maximize burst absorption
• Dynamic Packet Prioritization (DPP) – Prioritizes short-lived flows to
expedite flow setup and completion
Miercom Report: Speeding Applications in Data Centre Networks

http://miercom.com/cisco-systems-speeding-applications-in-data-center-networks/
Dynamic Buffer Protection (DBP)
• Prevents any output queue from consuming more than its fair share of
buffer in shared-memory architecture
• Defines dynamic max threshold for each queue
• If queue length exceeds threshold, packet is discarded
• Otherwise packet is admitted to queue and scheduled for transmission
• Threshold calculated by multiplying free memory by configurable, per-
α
queue Alpha (α) value (weight)
• Alpha controls how aggressively DBP maintains free buffer pages during
congestion events
Alpha Parameter Examples Default Alpha on
Cloud Scale switches
Alpha (α) = 0.5 Alpha (α) = 1 Alpha (α) = 14

40 40 40
35 35 Buffer per queue == 35
30 30 free buffer 30
Buffer per queue ==
½ free buffer Buffer per queue ==
Buffer in MB
Buffer in MB
Buffer in MB
25 25 25
14 x free buffer
20 20 20
15 15 15
10 10 10
5 5 5
0 0 0
1 2 4 8 16 32 64 1 2 4 8 16 32 64 1 2 4 8 16 32 64
Number of Congested Queues Number of Congested Queues Number of Congested Queues
Buffer per queue (MB) Free buffer (MB) Buffer per queue (MB) Free buffer (MB) Buffer per queue (MB) Free buffer (MB)
Buffering – Ideal versus Reality
Ideal buffer state Actual buffer state
Buffer available for burst absorption
Buffer available for burst absorption

Buffer consumed by sustained-
bandwidth TCP flows
Buffer consumed by sustained-

bandwidth TCP flows
Sustained-bandwidth TCP flows

Sustained-bandwidth TCP flows
consume all available buffer before
back off before all buffer
backing off
consumed
Approximate Fair Drop (AFD)
Maintain throughput while minimizing buffer consumption by elephant flows – keep buffer
state as close to the ideal as possible
Higher-bandwidth elephants – higher AFD drop
probability
Lower-bandwidth elephants – lower AFD drop probability
Distinguish elephant
flows from other flows
Queue admission
Ideal depth AFD

exceeded? Y set? N
Y
Buffer
Non-elephants – no AFD
Dynamic Packet Prioritization (DPP)
• Prioritize initial packets of new / short-lived flows
• Up to first 1023 packets of each flow assigned to higher-priority qos-
group
<= 1023 packets
Identify unique flows
Drive new, higher
priority
Track per-flow SP queue

packet count
Maintain original
priority
> 1023 packets Q-default

Ingress QOS / Egress Queuing Policies
• Default QOS behaviour: • To set/change packet markings,
• Trust received QOS markings use “set cos / precedence /
• All user data goes to q-default
dscp” in ingress QOS policy
• To select egress queue, use “set • To change queuing behaviour,

qos-group” in ingress QOS policy manipulate egress queuing
policies
Ingress QOS policy Egress Queuing policy
set qos- Selects queue-limit

Ingress
group queue priority
bandwidth
set dscp AFD
shape Egress
Marks
set prec
packet
set cos
Putting it all together
Create class-map
Attach policy-map class-map type qos class_foo
type qos and match cos 3-4
match on queuing to
cos/dscp/acls interface policy-map type qos pm1
class type qos class_foo
set qos-group 1
police cir 20 mbytes conform transmit violate drop
class type qos class-default
set qos-group 0
Create policy-map
Create policy-map
type qos and set interface ethernet 1/1
type queuing and service-policy type qos input pm1
qos-group and/or
create actions
add policing rule
class-map type queuing class-foo
match qos-group 1
policy-map type queuing policy-foo

class type queuing class-foo
Create class-map bandwidth percent 20
Attach policy-map class type queuing class-default
type queuing and
type qos as input to bandwidth percent 80
an interface
match on qos-
group interface ethernet 1/3
service-policy type queuing output policy-foo
Nexus 9000 QoS Golden Rules
• QoS is enabled by default and cannot
be disabled
• CoS and DSCP are TRUSTED by default
• Use QoS-Groups to tie policies together
• Queuing and QoS policies are applied to a
physical interface or at system level
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
M1 M2 LC
1G / 10G 10G / 40G / 100G
M3 delivers best of M-
and F-series capabilities
M-Series Modules
L2/L3/L4 with large forwarding tables and rich feature set
M3
10G / 40G / 100G
F1 F2/F2E
10G
10G
F3
10G / 40G / 100G
F4 increases
F-Series Modules
High performance, low latency with streamlined feature set
F3 closes the
F/M feature gap!
F3
100G port density
F4
F2E 100G
10G
10G / 40G / 100G
F3/M3/F4 – Ingress Buffered
Central Arbiter
Ingress Buffering Multistage Crossbar Fabric

Egress buffer – Receives frames
from fabric and schedules traffic
toward egress ports
EGRESS QUEUING POLICY

Independent q1
scheduling for each VOQ buffer
q2
destination/priority carved by source
q3
q4 and priority
4/8 priority levels
e2/9… per port (VQI)
e1/25
SP
DWRR
Virtual queuing – Virtual Ingress
Congestion management Queuing Buffer e2/9
Ingress buffer –
and local scheduling toward Buffers traffic for Egress Buffer
egress destinations (VQIs) congested egress
destinations (VQIs)
NETWORK-QOS POLICY
INGRESS QUEUING POLICY
F3/M3 I/O Module Buffering Capacity
Ingress
Module Total VOQ Buffer Ingress Queue Structure Ingress VOQ Buffer
Per Module
M3 48-port 10G 1500MB 4q1t 31.25MB / port
F3 48-port 10G 72MB 4q1t 1.5MB / port
M3 24-port 40G 3000MB 4q1t 125MB / port
F3 24-port 40G 144MB 4q1t 6MB / port
Egress
Module Egress VOQ Structure Egress VOQ Buffer (Credited) Egress VOQ Buffer (Uncredited)
M3 48-port 10G 1p7q1t 512KB / port 4MB / 24 ports
F3 48-port 10G 1p7q1t 295KB / port 512KB / 8 ports
M3 24-port 40G 1p7q1t 2MB / port 4MB / 6 port
F3 24-port 40G 1p7q1t 1.1MB / port 512KB / 2 ports
Ingress Queuing – Template View
8e Template 8e-4q4q Template 7e Template 6e Template 4e Template
CoS 5-7
CoS 5-7 CoS 5-7 CoS 5-7 CoS 5-7 (Q1)
(Q1) (Q1) (Q1) (Q1)
CoS 0
(Q-Default)
CoS 3-4 CoS 2,4 CoS 0-2
(Q3) (Q3) (Q-Default)
CoS 4
(Q3)
CoS 0-4 CoS 2 CoS 0,1 CoS 4
(Q-Default) (Q4) (Q-Default) (Q3)
Legend:
CoS 1-3 Drop Queue
CoS 0-1 CoS 3 CoS 3 (Q4)
(Q-Default) (Q4) (Q4) No-Drop Queue
Pause Active
High (Pause) Low (Resume)

Threshold Threshold
Egress Queuing – Logical View
default-4q-8e-out-policy default-4q4q-8e-out-policy default-4q-7e-out-policy default-4q-6e-out-policy default-4q-4e-out-policy
PQ1 Q2 Q3 Q-Def. PQ1 Q2 Q3 Q-Def. Q2 PQ1 Q3 Q-Def. PQ2. PQ3 PQ1 Q-Def. PQ1 Q-Def. PQ2 Q3
(5,6,7) (3,4) (2) (0,1) (5,6,7) (3,4) (2) (0,1) (3) (5,6,7) (2,4) (0,1) (4) (3) (5,6,7) (0-2) (5,6,7) (0) (4) (1,2,3)
DWRR DWRR DWRR
Priority Priority Priority Prio Prio Prio Prio
DWRR DWRR DWRR DWRR DWRR DWRR
PQ1 (H)
Q2
PQ2 (L)
PQ2 (H)
Egress Port
Egress Port
Egress Port
Egress Port
Egress Port
PQ3 (L)
PQ1
PQ1
PQ1
PQ1
Q-Def
Q-Def
Q-Def
Q-Def
Q-Def
Q3
Q2
Q3
Q2
Q3
Q3
33% 33% 33% 33% 33% 33% 50% 50% 100% 100% 100%
Green indicates no-drop

DSCP to CoS / CoS to DSCP – Mapping Tables
N7k# show table-map | grep -a 2 dscp- N7k# show table-map | grep -a 2 cos-
cos-map dscp-map
Table-map dscp-cos-map Table-map cos-dscp-map
default copy default copy
N7k# show system internal ipqos global- N7k# show system internal ipqos global-
defaults | grep -a 12 cos-dscp-map defaults | grep -a 12 dscp-cos-map
table-map: cos-dscp-map (len: 12) table-map: dscp-cos-map (len: 12)
default copy default copy
Bit array: Bit array:
Values set: Values set:
0 8 16 24 32 40 48 56 0 0 0 0 0 0 0 0
-- -- -- -- -- -- -- -- 1 1 1 1 1 1 1 1
-- -- -- -- -- -- -- -- DSCP 24-31 2 2 2 2 2 2 2 2
CoS 2 -- -- -- -- -- -- -- -- mapped to 3 3 3 3 3 3 3 3
--
mapped to -- -- -- -- -- -- -- CoS 3 4 4 4 4 4 4 4 4
--
DSCP 16-23 -- -- -- -- -- -- -- 5 5 5 5 5 5 5 5
-- -- -- -- -- -- -- -- 6 6 6 6 6 6 6 6
-- -- -- -- -- -- -- -- 7 7 7 7 7 7 7 7
Note: Output taken from Nexus 7000
CoS or DSCP to Queue Mapping
• Default CoS to Queue Mapping for Nexus 7000/7700 (F- and M-
Series I/O Module)
• Ingress: CoS to Queue
• Egress: CoS to Queue
• DSCP to Queue Mapping for Nexus 7000/7700 (F- and M-Series

I/O Module)
• Ingress: DSCP to Queue
• Egress: CoS to Queue
• Global Configuration (Admin/Default VDC) required to enable DSCP

to Queue Mapping:
N7k(config)# hardware qos dscp-to-queue ingress module type {all | f-series | m-series}
Changing the Default Trust
routed/bridged
802.1q Trunk 802.1q Trunk /

Access Port
Ingress Egress
Queues Queues
q-n
DSCP [24] DSCP [0]
CoS
CoS
L3 L3
data q-1 data
L2
CoS [2] L2
CoS [0]
q-default
Set CoS 0
DSCP
DSCP[24]
[0]
(will only set CoS to 0) data L3
L3
L2
CoS [0] Note: CoS is used for egress
queue selection, even if the
egress interface does NOT
Set DSCP 0 carry CoS in the frame
(will set DSCP + CoS to 0)
Default Rules Summary
Routed Traffic Bridged Traffic
• If CoS and DSCP is present • If CoS and DSCP is present
• CoS is used for ingress queue selection • CoS is used for ingress queue selection
• DSCP is preserved and rewrites CoS (top • CoS is preserved
most 3bit) • DSCP is unmodified
• CoS is used for egress queue selection • CoS is used for egress queue selection
• If only DSCP is present • If only DSCP is present
• No CoS gets treated as CoS 0 on ingress • No CoS gets treated as CoS 0 on ingress
• DSCP is preserved and rewrites CoS (top • CoS 0 is used for ingress and egress
most 3bit) queue selection
• CoS (derived from DSCP) drives egress • DSCP is unmodified
queue selection
be disabled
• Default Queuing and QoS policies are applied
to all physical interfaces across all VDCs
• For bridged traffic, CoS is preserved, DSCP
is unmodified
• For routed traffic, DSCP is copied to CoS
(first 3 bits)
• Ex: DSCP 40 (b101000) becomes CoS 5 (b101)
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
Nexus 5000 Series Overview
10G\40G Scalability Density
(Large Buffers
Nexus 5600 and Tables) Programmability
High 10G Density Network Visibility
CISCO INNOVATION
Fabric Innovations
CUSTOMER VALUE
40G Flexibility
100G Uplinks
Unified Ports VXLAN
Buffers/Tables
Nexus 5500 LAN/SAN Convergence FabricPath

Flexibility
10G Uplinks 20K+ Customers
Over 5 Years
25M+ Ports Shipped
Nexus 5010/5020 FEX Architecture 125K+ Chassis Shipped

75%+ Market Share*
Key Concepts – Common Points
Nexus 7000 compared to Nexus 5000 QoS
• Nexus 5000/6000 and Nexus 7000 F-
Series I/O Modules share the Ingress Buffer
Model
• Ingress buffering and queuing occur at VOQ
of each ingress port
• Egress scheduling enforced by egress port
• No Egress QOS Policies
• Packet marking
• DCBX 802.1Qaz
• Ingress policing (No egress policing)

• Flexible buffer management
Cisco Nexus 5672UP Internal Architecture
Fabric
Supervisor
UPC 1 UPC 2 UPC 3 UPC 4 UPC 5 UPC 6 UPC-0

FC PHY
24xSFP+ 24xSFP+ 6xQSFP+ CPU
Slot 1 Slot 2
Packet Buffering
• 25MB packet buffer is shared by every three 40 GE ports or twelve 10 GE ports.
• Buffer is 16MB at ingress and 9MB at egress.
• Unicast packet can be buffered at both ingress and egress.
• Multicast Buffered at egress only
Ingress
Unicast VOQ
Egress UPC
UPC
9MB
16MB 224
448
Gbps Unified Gbps
Crossbar
Multicast VOQ
Fabric
Flexible Buffer Management
Ingress Buffer
• Shared buffer is good for burst UPC Ingress Buffer (16MB)
absorption.
• Dedicated buffer is good for SPAN Control
predictable performance for each
port.
• On by default, no configuration Shared Packet Buffer
needed
Port 1 Port 2 Port 3
• Long-distance FCoE, video Dedicated Dedicated Dedicated
editing (i.e., AVID), Big Data, and
distributed storage
Default Ingress Buffer Allocation
• Each cell is 320 bytes.
• Total number of cells for ingress buffer is 48,840.
Buffer Pool 10 GE Port 40 GE Port
Control traffic (per port) 64 KB 67.2 KB
SPAN (per port) 38.4 KB 153.6 KB
Class default (per port) 100 KB 100 KB
Shared buffer 13.2 MB 14.7 MB
Tune Buffer Allocation at Ingress
• “queue-limit” under “network-qos” policy specifies the dedicated buffer for each port and each
class. The dedicated buffer can be used by the port for only that class of service.
• Without “queue-limit” each class of service will get 100 KB of dedicated buffer.
• The size of dedicated buffer can be different for different classes of service. The policy applies
to all ports in the chassis.
• Total ingress buffer minus the dedicated buffer and buffer for control and SPAN will be in the
shared buffer pool.
• The following example sets the dedicated buffer for “class-default” to be 400 KB for all ports.
switch(config)# policy-map type network-qos Policy-buffer

switch(config-pmap-nq)# class type network-qos class-default
switch(config-pmap-nq-c)# queue-limit 400000 bytes
switch(config-pmap-nq-c)# system qos
switch(config-sys-qos)# service-policy type network-qos Policy-buffer
Flexible Buffer Management
Egress Buffer
• 9-MB packet buffer is shared
UPC Egress Buffer (9MB)
among three 40 GE or twelve
10 GE.
• CLI is provided to allocate buffer
Unicast Buffer
from unicast to multicast.
• Unicast traffic can be buffered at
egress and ingress.
Multicast Buffer
• Multicast is buffered at egress in
case of interface
oversubscription.
Default Egress Buffer Allocation
• Software provides CLI to tune the egress buffer allocation.
• At egress, unicast buffer is allocated on a per-port basis. For
multicast, the egress buffer is shared among all ports.
• Use ”hardware multicast-buffer-tune” to assign unicast buffer to
multicast pool on egress
Buffer pool 10GE Port 40GE Port
650KB with 10G fabric mode
Unicast (per port) 363 KB
635KB with 40G fabric mode
Multicast (per ASIC) 4.3 MB 6.6 MB
Nexus 5600/6000 QoS Configuration Model
• Uses QOS-Groups to tie together QoS,
Queuing and Network-QoS policies
• QoS-Group has no direct relation with
priority values
• QoS-Groups defined (set) in policy-map
type qos.
• QoS-groups referenced (match) in policy
type queuing and policy-map type
network-qos
Putting it all together
class-map type qos class_foo
Create class-map match cos 3-4
type qos and Attach policy-map
queuing to policy-map type qos pm1
match on class type qos class_foo
cos/dscp/acls interface
set qos-group 1
class type qos class-default
set qos-group 0
interface ethernet 1/1

service-policy type qos input pm1
Create policy-map
Create policy-map
type qos and set class-map type queuing class-foo
type queuing and
qos-group and/or match qos-group 1
create actions
add policing rule
policy-map type queuing policy-foo
class type queuing class-foo
bandwidth percent 20
class type queuing class-default
Create class-map
Attach policy-map
type queuing and interface ethernet 1/3
type qos as input to
match on qos- service-policy type queuing input policy-foo
an interface
group
Buffering Capacity
Ingress
Traffic Type Ingress Queue Structure 10 GE Port 40 GE Port
Control traffic (per port) 6q1t 64 KB 67 KB
6q1t
Span Traffic (per Port) 38.4 KB 154 KB
6q1t
Class Default (per Port) 100 KB 100 KB
6q1t
Shared Buffer 13.2 MB 14.7 MB
Egress
Traffic Type Egress Queue Structure 10 GE Port 40 GE Port
1p5q0t 650 KB with 10GB Fabric Mode

Unicast 363 KB
635 KB with 40GB Fabric Mode
1p5q0t
Multicast 4.3MB 6.6 MB
• WRED is enabled by default and cannot be disabled
• Use QoS-Groups to tie policies together
• No Egress QOS policies
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
Nexus 3000 Series Switches
Nexus 3100 Nexus 3200 Nexus 3600

• ToR Leaf • Fixed High Density • Deep Buffer
• Full-featured DC access • High throughput and performance • High route scale
• Broad switch portfolio • Flexible connectivity options • Video and Drop sensitive
• Based on Trident ASIC family • Based on Tomahawk ASIC family deployments
• Based on Jericho ASIC family
Nexus 3400 Nexus 3500

• Programmable pipeline • Ultra Low Latency
• Support for P4-INT • Financial/HFT workloads
• Enable custom use cases • Based on Cisco Monticello ASICs
• Includes Tofino and Teralynx ASICs
• Packet marking
• Tail Drop and WRED with ECN
• Shared buffer capability
• Egress Queuing
• 3-level hierarchical scheduling
Hardware Scheduler Implementation
• 3 level scheduling hierarchy
S3 S2 S1
Control
traffic
UC0
UC1
UC winner
UC7
MC0 MC winner
MC3
Dynamic Buffer Protection
• Buffer is shared dynamically any queue
can use shared buffer
• Dynamic Buffer Protection prevents any
queue unfair use shared buffer
• The basic algorithm uses dynamic queue
length threshold, and account for usage of
unicast and multicast
be disabled
• Use QoS-Groups to tie policies
together
• Queuing and QoS policies are applied
to a physical interface or at system
level
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
FEX Overview
• Scalable and Extensible
Fabric
• Single point of management
• Homogeneous and consistent
policies
By Author listed as "U.S. Air Force photo" [Public domain], via Wikimedia Commons
• DSCP, CoS
• ACL classification (FEX offload)
on
Nexus 5600/6000
• Strict Priority Queuing and
DWRR
• Queue-limit Carving
FEX Policy Offload (Nexus 5600/6000 only)
• TCAM resources on a FEX to perform ACL-based classification
• The feature is disabled by default
• By default, a FEX classifies packets on CoS value
• Both system level and interface level policies are offloaded to the
FEX
switch# configure terminal

fex chassis_ID
hardware card-type qos-policy-offload
FEX Policy with Nexus 9000 as parent
• The FEX QoS policy is applied to the hardware resources of the
fabric port associated with the FEX HIF port
• Classification is based on the COS value.
• System level input queueing for DWRR and Strict priority scheduling
for HIF to NIF traffic and for NIF to HIF traffic
• Queuing:
• 4 queues are present on the FEX
• The scheduling is done per port and each port has its own scheduler.
FEX Queuing Policies – Nexus 7000
• On Nexus 7000 with FEX + M-Series parent modules, network-qos
and F-series ingress queuing class-maps drive FEX queuing
configuration
• Ingress queuing class-maps drive:
• Both ingress and egress COS/DSCP-to-queue mapping
• Enabling DSCP-to-queue on parent switch enables DSCP-to-
queue on FEX
• DSCP-to-queue only active in the HIF→NIF direction
• NIF→HIF direction always uses COS-to-queue mapping, based on COS
transmitted by parent switch to FEX
FEX Queue-Limit – Nexus 7000
• Provides FEX queue-limit configuration option
• Manages buffer thresholds on FEX based on platform capabilities
• Default has queue-limit enabled
• Configuration applied per-VDC (on Nexus 7000/7700)
• Different FEX models have different capabilities
• FEX QOS classification on COS or DSCP
unless FEX offload enabled
• FEX queuing driven implicitly by parent
switch queuing configuration
• No support for per-queue shaping, policing
or marking
• Drop thresholds are tail-drop only, no WRED
support
Agenda
• Introduction
• QoS Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
What do we want to achieve?
Company XYZ’s Business Goals
• Make sure no disruption in network services

• Put control traffic in priority queue
• Video/voice hosting also an business objective

• Put voice traffic in priority queue
• Dedicated bandwidth to video traffic
• Flexibility in moving applications across servers

• Dedicated bandwidth to vmotion/mobility
• Everything else best-effort
Translating to the language of QoS
Queuing Queue-Limit
Application CoS Character
(Scheduling) (Buffer)
BW remaining High Volume /
Best Effort 0, 1 60%
50% Less Important
vMotion / Live BW remaining Medium Volume /
2 10%
Migration 20% Important
BW remaining Medium Volume
Multimedia 3, 4 20%
30% Very Important
Low Volume /
Strict Priority 5 Important /
Priority Queue 10% Delay Sensitive
Low Volume /
Network Control 6,7
Very important
Topology
M3 cards facing
Core
core and M3/F3
cards facing Nexus 7000 Nexus 7000
access
vpc peer-link
VPC from
access to
aggregation
Nexus 9000 Nexus 9000 Nexus 5000 Nexus 5000 Nexus 9000 Nexus 9000
vpc peer-link vpc peer-link vpc peer-link
Nexus 2000 Nexus 2000 Nexus 2000 Nexus 2000 Nexus 2000 Nexus 2000
Host VPC,
Enhanced VPC Straight-through
Straight-through
(evPC) FEX, No VPC
FEX
Type:
Classification, Marking and Trust on Nexus 5000/7000/9000 QoS
Core
Mark Traffic from Core

(policy-map type qos)
or just TRUST (default) vpc peer-link Between the
different Tiers,
all ports are
TRUSTED

Mark Traffic
from Servers
(policy-map
type qos)
or just TRUST
(default)
Type:
QoS
Classification and Marking: Nexus 7000

ip access-list ACL_QOS_LOWPRIO policy-map type qos PM_QOS_MARK_COS_IN
10 permit … class CM_QOS_STRICTPRIO_COS5
ip access-list ACL_QOS_VMOTION set cos 5
10 permit … class CM_QOS_MULTIMEDIA_COS4
ip access-list ACL_QOS_MULTIMEDIA set cos 4
10 permit … class CM_QOS_VMOTION_COS2
ip access-list ACL_QOS_STRICTPRIO set cos 2
10 permit … class CM_QOS_LOWPRIO_COS1
! set cos 1
class-map type qos match-any CM_QOS_LOWPRIO_COS1 !
match access-group name ACL_QOS_LOWPRIO interface Ethernet1/1
! service-policy type qos input PM_QOS_MARK_COS_IN
class-map type qos match-any CM_QOS_VMOTION_COS2 !
match access-group name ACL_QOS_VMOTION vlan configuration 100
! service-policy input PM_QOS_MARK_COS_IN
class-map type qos match-any CM_QOS_MULTIMEDIA_COS4
match access-group name ACL_QOS_MULTIMEDIA
!
class-map type qos match-any CM_QOS_STRICTPRIO_COS5
match access-group name ACL_QOS_STRICTPRIO
Type:
QoS
Classification and Marking: Nexus 5600 (1)

ip access-list ACL_QOS_VMOTION set qos-group 5
10 permit … class CM_QOS_MULTIMEDIA_COS4
ip access-list ACL_QOS_MULTIMEDIA set qos-group 4
10 permit … class CM_QOS_VMOTION_COS2
! set qos-group 3
class-map type qos match-any CM_QOS_LOWPRIO_COS1 class CM_QOS_LOWPRIO_COS1
match access-group name ACL_QOS_LOWPRIO set qos-group 2
! !
class-map type qos match-any CM_QOS_VMOTION_COS2 system qos
match access-group name ACL_QOS_VMOTION service-policy type qos input PM_QOS_MARK_COS_IN
!
class-map type qos match-any CM_QOS_MULTIMEDIA_COS4
match access-group name ACL_QOS_MULTIMEDIA
!
match cos 5
QoS-Group # is mapping between Slide 1 and Slide 2
Type:
Networ-QoS
Classification and Marking: Nexus 5600 (2)

class-map type network-qos CM_N-QOS_MATCH_QG2_COS1 policy-map type network-qos PM_N-QOS_SYSTEM
match qos-group 2 class type network-qos CM_N-QOS_MATCH_QG2_COS1
class-map type network-qos CM_N-QOS_MATCH_QG3_COS2 set cos 1
set cos 5
queue-limit 20480 bytes
!
system qos
service-policy type network-qos PM_N-QOS_SYSTEM
QoS-Group # is mapping between Slide 1 and Slide 2
Type:
QoS
Classification and Marking: Nexus 9000

ip access-list ACL_QOS_VMOTION set qos-group 5
10 permit … set cos 5
ip access-list ACL_QOS_MULTIMEDIA class CM_QOS_MULTIMEDIA_COS4
10 permit … set qos-group 4
! set cos 4
class-map type qos match-any CM_QOS_LOWPRIO_COS1 class CM_QOS_VMOTION_COS2
match access-group name ACL_QOS_LOWPRIO set qos-group 3
! set cos 2
class-map type qos match-any CM_QOS_VMOTION_COS2 class CM_QOS_LOWPRIO_COS1
match access-group name ACL_QOS_VMOTION set qos-group 2
! set cos 1
class-map type qos match-any CM_QOS_MULTIMEDIA_COS4 !
match access-group name ACL_QOS_MULTIMEDIA system qos
! service-policy type qos input PM_QOS_MARK_COS_IN
match cos 5
Network-QoS Configuration on M3/F3-Series
Core
Use the network-

qos policy for 8e-
vpc peer-link
4q4q i.e 4
ingress queues
Type:
Network-QoS
Network-QoS Configuration –M3/F3 cards

Example (Admin- / Default-VDC)
Core
system qos
service-policy type network-qos default-nq-8e-4q4q-policy
Admin- vpc peer-link Admin-
policy-map type network-qos default-nq-8e-4q4q-policy template 8e-4q4q VDC VDC
class type network-qos c-nq-8e-4q4q
match cos 0-7
congestion-control tail-drop
mtu 1500
vpc peer-link
Changes apply to ALL ports of specified type in ALL VDCs

Changes are traffic disruptive for ports of specified type
Queuing (M3/F3 cards)
Type:
Queuing
Core Modify CoS to Queue

mapping in Admin-
Assign a egress policy- /Default-VDC
map to each interface with
priority, bandwidth and vpc peer-link
queue-limit to alter default
queuing policy Assign a ingress policy-map
for buffer allocation, no
ingress scheduling
CoS to Queue Mapping – M3/F3 I/O Module
Example
Queuing
Queue-Limit Queue
Application CoS (Scheduling)- Character
(Buffer)-ingress (Ingress/Egress)
egress
4q1t-8e-4q4q-in-q-default
0,1 BW remaining High Volume /
Best Effort 50% / 1p3q1t-8e-4q4q-out-q-
50% Less Important
default
vMotion / Live BW remaining 4q1t-8e-4q4q-in-q4 / Medium Volume /

2 10%
Migration 20% 1p3q1t-8e-4q4q-out-q3 Important
BW remaining 4q1t-8e-4q4q-in-q3 / Medium Volume
Multimedia 3, 4 30%
30% 1p3q1t-8e-4q4q-out-q2 Very Important
Low Volume /
4q1t-8e-4q4q-in-q1 / Delay Sensitive
Priority Queue 10%
1p3q1t-8e-4q4q-out-pq1
Low Volume /
Network Control 6/7
Very important
CoS to Queue Configuration –M3/F3 slides
Type:
Queuing
Example (Admin- / Default-VDC)

class-map type queuing match-any 4q1t-8e-4q4q-in-q1 Core
match cos 5-7
class-map type queuing match-any 4q1t-8e-4q4q-in-q-default
match cos 0-1 Admin- Admin-
vpc peer-link
class-map type queuing match-any 4q1t-8e-4q4q-in-q3 VDC VDC
match cos 3-4
class-map type queuing match-any 4q1t-8e-4q4q-in-q4
match cos 2
class-map type queuing match-any 1p3q1t-8e-4q4q-out-pq1 vpc peer-link

match cos 5-7
class-map type queuing match-any 1p3q1t-8e-4q4q-out-q2
match cos 3-4
class-map type queuing match-any 1p3q1t-8e-4q4q-out-q3
match cos 2
class-map type queuing match-any 1p3q1t-8e-4q4q-out-q-default
match cos 0-1
Changes apply to ALL ports of specified type in ALL VDCs

Changes are traffic disruptive for ports of specified type
Ingress Queuing Configuration for M3/F3 cards
Type:
Queuing
Example (Payload-VDC)
qos copy policy-map type queuing default-8e-4q4q-in-policy prefix Core
Custom-
policy-map type queuing Custom-8e-4q4q-in

class type queuing 4q1t-8e-4q4q-in-q1 vpc peer-link
queue-limit percent 10
class type queuing 4q1t-8e-4q4q-in-q-default
class type queuing 4q1t-8e-4q4q-in-q3
queue-limit percent 30 vpc peer-link
class type queuing 4q1t-8e-4q4q-in-q4
interface Ethernet1/1
service-policy type queuing input Custom-8e-4q4q-in
All Policy-Map and Service-Policy are done in relevant Payload-VDC and

only affect the interface to which they get applied
Egress Queuing Configuration for M3/F3 cards
Type:
Queuing
Example (Payload-VDC)
Core
qos copy policy-map type queuing default-8e-4q4q-out-policy prefix
Custom-
vpc peer-link
policy-map type queuing Custom-8e-4q4q-out
class type queuing 1p3q1t-8e-4q4q-out-pq1
priority level 1
class type queuing 1p3q1t-8e-4q4q-out-q2
bandwidth remaining percent 30
class type queuing 1p3q1t-8e-4q4q-out-q3
bandwidth remaining percent 20 vpc peer-link
class type queuing 1p3q1t-8e-4q4q-out-q-default
!
service-policy type queuing output Custom-8e-4q4q-out
All Policy-Map and Service-Policy are done in relevant Payload-VDC and

only affect the interface to which they get applied
CoS to Queue Mapping - Nexus 9000
Example
Queuing Queue limit Queue

Application CoS Character
(Scheduling) (Alpha) (6q1t / 1p6q0t)
BW percent High Volume /
Best Effort 0,1 Default (9) qos-group 0 (default)
40% Less Important
vMotion / Live BW percent Medium Volume /
2,3 Default (9) qos-group 3
Migration 20% Important
BW percent Medium Volume
Multimedia 4 Default (9) qos-group 4
30% Very Important
Low Volume /
BW percent Delay Sensitive
Default (9) qos-group5 / priority
10%
Low Volume /
Network Control 6,7
Very important
Type:
Queuing
Egress Queuing Configuration: Nexus9000

Example
class-map type queuing CM_Q_MATCH_QG3_COS2 Core
match qos-group 3
class-map type queuing CM_Q_MATCH_QG4_COS4
match qos-group 4
class-map type queuing CM_Q_MATCH_QG5_COS5 vpc peer-link
match qos-group 5
!
policy-map type queuing PM_QUEUING_SYSTEM_OUT
class type queuing CM_Q_MATCH_QG3_COS2
vpc peer-link
priority
Queuing Nexus 2000
Type:
Queuing
Core
vpc peer-link
Queuing on
vpc peer-link NIF (multiple vpc peer-link vpc peer-link
no-drop
queues) only
available
with CoS
Queuing on
based
NIF
marking on
controlled
HIF
by Fabric
Interface
INPUT
policy
Type:
Queuing
Queuing Configuration (Nexus 2000)

Example
class-map type queuing CM_Q_MATCH_QG3_COS2 Core
match qos-group 3
class-map type queuing CM_Q_MATCH_QG4_COS4
match qos-group 4
class-map type queuing CM_Q_MATCH_QG5_COS5 vpc peer-link
match qos-group 5
!
policy-map type queuing PM_QUEUING_SYSTEM_N2K
vpc peer-link
priority
Amount of Queues depend on FEX (Nexus 2000) Model

Agenda
• Introduction
• QoS and Queuing Basics
• Nexus 9000 QoS
• Nexus 7000/7700 QoS
• Nexus 5600 QoS
• Nexus 3000 QoS
• Nexus 2000 QoS
• Conclusion
Why QoS in the Data Centre?
Assign Manage Maximise
Colour to Traffic Congestion Throughput
Maximise Throughput and Manage Congestion!
Recommended Reading
• End-to-End QoS Network Design:
Quality of Service for Rich-Media
and Cloud Networks, 2nd Edition
• Tim Szigeti
• Christina Hattingh
• Robert Barton
• Kenneth Briley
• ISBN-10: 1-58714-369-0
• ISBN-13: 978-1-58714-369-4
With some help of my friends
I would like to thank all the people, who started
the QoS journey and contributed to it:
• Lukas Krattiger, Principal Engineer
• Tim Stevenson, Distinguished Technical
Marketing Engineer
• Matthias Wessendorf,
Technical Marketing Engineer
Bonus Slides
FEX QoS
Configuration
Examples
Fex QoS Policy Configuration Example
policy-map type qos fex-qos
class fex-qos-class-1
set dscp 10
Marking policy
set dscp 18
set dscp 26
Policy applied on ingress of FEX HIF
!
interface Ethernet101/1/1
service-policy type qos input fex-qos
Nexus 7000 Network-QoS Configuration Example #1
• Applying 8e-4q4q template to enable 4 ingress/egress queues on FEX with COS to
queue mapping (also enables 4 ingress queues on F-series modules, if present)
system qos Default 8e-4q4q template applied to
service-policy type network-qos default-nq-8e-4q4q-policy “system qos” target
• FEX output (“show queuing interface”):

Queuing:
queue qos-group cos priority bandwidth mtu
--------+------------+--------------+---------+---------+----
ctrl-hi n/a 7 PRI 0 2400
ctrl-lo n/a 7 PRI 0 2400
2 0 0 1 WRR 30 1600
3 1 2 WRR 30 1600
8e4q4q configuration (4 data traffic queues)
4 2 5 6 WRR 10 1600
5 3 3 4 WRR 30 1600
Nexus 7000 Network-QoS Configuration Example #2
• Applying custom 8e-4q4q-based template with new MTU
policy-map type network-qos custom-nq-8e-4q4q template 8e-4q4q
class type network-qos c-nq-8e-4q4q Custom network-qos policy
congestion-control tail-drop with new MTU
mtu 9216
system qos Custom template applied to
service-policy type network-qos custom-nq-8e-4q4q “system qos” target
• FEX output (“show queuing interface”) after MTU change:

Queuing:
queue qos-group cos priority bandwidth mtu
--------+------------+--------------+---------+---------+----
ctrl-hi n/a 7 PRI 0 2400
ctrl-lo n/a 7 PRI 0 2400
2 0 0 1 WRR 30 9280
3 1 2 WRR 30 9280
MTU increased on data traffic queues
4 2 5 6 WRR 10 9280
5 3 3 4 WRR 30 9280
Modifying CoS- or DSCP-to-Queue Mappings
• Changing CoS- or DSCP-to-queue mappings in parent switch F-type
ingress queuing class-maps modifies mappings on FEX
• Queuing class-maps modified only in default/admin VDC (apply to entire
system) Queuing:
Non-default F-series queue qos-group cos priority bandwidth mtu
class-map type queuing match-any 4q1t-8e-4q4q-in-q1 ingress queuing
--------+------------+--------------+---------+---------+----
match cos 1-3 class-maps (COS and ctrl-hi n/a 7 PRI 0 2400
match dscp 8-31 DSCP match statements ctrl-lo n/a 7 PRI 0 2400
class-map type queuing match-any

modified)
4q1t-8e-4q4q-in-q-default 2 0 0 WRR 30 1600
3 1 6 WRR 30 1600
match cos 0
4 2 1 2 3 WRR 10 1600
match dscp 0-7
5 3 4 5 WRR 30 1600
class-map type queuing match-any 4q1t-8e-4q4q-in-q3 <…>
match cos 4-5
queue DSCPs
match dscp 32-47
----- -----
class-map type queuing match-any 4q1t-8e-4q4q-in-q4
FEX queue mappings 02 0-7,
match cos 6-7
reflect changes 04 8-31,
match dscp 48-63 03 48-63,
05 32-47,
Enabling FEX Queue Limits
• Example #1 – N2K-C2248TP-1GE
fex 101
hardware N2248T queue-limit 50000
• Example #2 – N2K-C2232TM-E-10GE
fex 102
hardware N2232TM-E queue-limit 50000
• FEX output (“show queuing interface”) before:

Queue limit: Disabled
• FEX output (“show queuing interface”) after (configured queue-limit rounded to nearest
hardware supported value):
Queue limit: 51200 bytes
F3 Queuing
Configuration
Examples
Central Arbiter
F3 10G Buffering Model

Virtual Queuing
8K VOQs Scheduling
(1K Dest + 8 priority levels)
Virtual Queuing / Egress Buffer – Receives
frames from fabric; Schedules frame transmission
EGRESS QUEUING POLICIES

8 priority
4 ingress queues levels
per port
Ingress Buffer Egress Buffer
DWRR
10G Port 1 1.5MB VQI 1 10G Port 1
SP
FABRIC
Buffering / queuing
…
Ingress Buffer – Manages congestion toward
…
egress destinations
INGRESS QUEUING POLICIES
DWRR
VQI 8 10G Port 8
10G Port 8 1.5MB
SP
Egress Scheduling
Ingress SOC Egress SOC

Central Arbiter

Virtual Queuing
8K VOQs Scheduling
(1K VQIs + 8 priority levels)
8 priority
4 ingress queues levels
per port Ingress Buffer Egress Buffer
DWRR
40G Port 1 6MB VQI 1 40G Port 1
FABRIC
SP
Buffering / queuing
egress destinations (VQIs)
DWRR
INGRESS QUEUING POLICIES VQI 2 40G Port 2
40G Port 2 6MB
SP
Egress Scheduling

Central Arbiter

Virtual Queuing
8K VOQs Scheduling
(1K VQIs + 8 priority levels)
4 ingress queues
per port Ingress Buffer Egress Buffer 8 priority
levels
DWRR
FABRIC
VQI 1 100G Port 1
100G Port 1
SP
12MB
Buffering / queuing
egress destinations (VQIs)
INGRESS QUEUING POLICIES

Egress Scheduling

Network-QoS and Default Queuing (global)
N7k# show policy-map type queuing | beg default 4q-8e
• default-nq-8e-policy is default network-
qos policy and attached to system qos policy-map type queuing default-4q-8e-in-policy
class type queuing 2q4t-8e-in-q1
in Admin-/Default-VDC queue-limit percent 10
• The system queuing policy applied by class type queuing 2q4t-8e-in-q-default
default can be overridden on a per port bandwidth percent 50
basis. policy-map type queuing default-4q-8e-out-policy
class type queuing 1p3q1t-8e-out-pq1
priority level 1
class type queuing 1p3q1t-8e-out-q2
class type queuing 1p3q1t-8e-out-q3
class type queuing 1p3q1t-8e-out-q-default
2 ingress queues
with buffer ratio 4 egress queues
1:9 and DWRR with one priority
wieights 1:1 queue and DWRR
wieights 1:1:1
Note: show policy-map system does display similar output

Modifying Queuing and Scheduling Behaviour on F3
Modules
I want to… Steps to follow
…remap COS/DSCP values from one 1. Modify the type queuing class-map(s) for the desired queue(s)
queue to another queue without activating
additional queues
…change queuing behaviour without 1. Define new type queuing policy-map (you cannot modify the
changing COS-or DSCP-to-queue default policies)
mapping 2. Modify class-map parameters
3. Apply new policy-map to interfaces
…activate additional queues and remap 1. Define new type queuing policy-map
COS/DSCP values 2. Modify COS-to-queue mapping for target port type
3. Apply new policy-map to interfaces
…shape the SP queue 1. (Optional) Clone the default egress queuing policy
2. Shape the SP queue in the new (cloned) policy
3. Apply the new queuing policy to the target interfaces
Modifying Queuing Behaviour
Remap Some COS/DSCP Values from One Queue to Another Queue without
Activating Additional Queues
Modify “type queuing” class-map(s) for desired

queue(s)
Remap COS- or DSCP-to-queue mapping for given queue(s)
Important: changing COS- or DSCP-to-queue mapping takes effect immediately and is disruptive
to all ports
Remap Some COS/DSCP Values from One Queue to Another Queue without Activating Additional
Queues
Example: remap COS 4 and DSCP 32-39 to ingress queue “q1”:
n77# show class-map type queuing 8q2t-in-q1
Type queuing class-maps
========================
Show current
class-map type queuing match-any 8q2t-in-q1 mapping
Description: Classifier for ingress queue 1 of type 8q2t
match cos 5-7
match dscp 40-63
n77# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.

n77(config)# ! Modify ingress queue q1
Configure new
n77(config)# class-map type queuing match-any 8q2t-in-q1
mapping
n77(config-cmap-que)# ! Change COS- and DSCP-to-queue mapping for this queue
n77(config-cmap-que)# match cos 4
n77(config-cmap-que)# match dscp 32-39
n77(config-cmap-que)# show class-map type queuing 8q2t-in-q1

Show new
Type queuing class-maps mapping
========================
class-map type queuing match-any 8q2t-in-q1
Description: Classifier for ingress queue 1 of type 8q2t
match cos 4-7
match dscp 32-63
n77(config-cmap-que)# #CLUS BRKDCN-3346 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 146
Changing Default Queuing Behaviour without Changing COS- or DSCP-to-
Queue
Define new “type queuing” policy-map for port

type
Create new policy
Modify class-map parameters
Define behaviour of each queue
Apply “type queuing” service-policy to

interface(s)
Apply new queuing policy to interface(s)
Important: applying new queuing policy takes effect immediately and is disruptive to any ports to
which the policy is applied
Changing Default Queuing Behaviour without Changing COS- or DSCP-to-
Queue
Example: Resize ingress queues without modifying COS- or DSCP-to-queue mapping
n77(config)# ! Define new "type queuing" policy Create new
n77(config)# policy-map type queuing new-f3-ingress queuing policy
n77(config-pmap-que)# ! Define behavior for F3 ingress q-default
n77(config-pmap-que)# class type queuing 8e-4q8q-in-q-default
n77(config-pmap-c-que)# ! Resize this queue
n77(config-pmap-c-que)# queue-limit percent 74
n77(config-pmap-c-que)# ! Define behavior for F3 ingress queue 1
n77(config-pmap-c-que)# class type queuing 8e-4q8q-in-q1 Modify class-map
parameters (resize queues)
n77(config-pmap-c-que)# ! Policy must include all queues (even inactive)
n77(config-pmap-c-que)# class type queuing 8e-4q8q-in-q3
n77(config-pmap-c-que)# ! Must give at least 1% to inactive queues
n77(config-pmap-c-que)# class type queuing 8e-4q8q-in-q4 Apply policy to
interface(s)
n77(config-pmap-c-que)# interface e 2/1-48
n77(config-if-range)# ! Apply the new policy to F3 interfaces
n77(config-if-range)# service-policy type queuing input new-f3-ingress
n77(config-if-range)# #CLUS BRKDCN-3346 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 148
Activate Additional Queues and Remap COS/DSCP Values
Modify “type queuing” class-map(s) for desired

queue(s)
Remap COS- or DSCP-to-queue mapping to additional queue(s)
Define new “type queuing” policy-map for port

type
Create new policy
Modify class-map parameters

Define behaviour of each queue
Apply “type queuing” service-policy to

interface(s)
Apply new queuing policy to interface(s)
Important: changing COS/DSCP-to-queue mapping and takes effect immediately and is disruptive to all ports;
applying new queuing policy takes effect immediately and is disruptive to any ports to which the policy is applied
Activate Additional Queues and Remap COS/DSCP Values
Example: Enable one additional ingress queue and map COS/DSCP values to all active queues
n77(config)# ! Modify ingress queue q3
Map COS/DSCP values to
n77(config)# class-map type queuing match-any 8q2t-in-q3 inactive queue
n77(config-cmap-que)# ! Map COS and DSCP values to this queue
n77(config-cmap-que)# match cos 3-4
Create new
n77(config-cmap-que)# ! Define new "type queuing" policy
n77(config-cmap-que)# policy-map type queuing new-f3-ingress queuing policy
n77(config-pmap-que)# ! Define behavior for F3 ingress q-default
n77(config-pmap-c-que)# queue-limit percent 60 Modify class-map
parameters (resize queues)
Apply policy to
n77(config-pmap-c-que)# ! Policy must include all queues (even inactive) interface(s)
n77(config-pmap-c-que)# int e 2/1-48
n77(config-if-range)# ! Apply the new policy to F3 interfaces
Important!
• If you change the COS- or DSCP-to-queue mapping for a port type, make sure all ports of that
type in all VDCs have a queuing policy applied that defines behaviour for all queues with
COS/DSCP values mapped
• For example, if you do THIS…
n77(config)# class-map type queuing match-any 8e-4q8q-in-q-default

Changes the default COS/
DSCP-to-queue mapping
n77(config-cmap-que)# class-map type queuing match-any 8e-4q8q-in-q4
n77(config-cmap-que)# match cos 5
Important!
Defines a new queuing policy that defines
• … then make sure you do THIS… behaviour of all queues that COS/DSCP
values have been mapped to
n77(config)# policy-map type queuing new-f3-ingress

n77(config-pmap-c-que)#
Important!
Maps the new policy to ALL interfaces in the
• … and then do THIS: system (do this on ALL ports in EVERY VDC!!)
n77(config)# int e 2/1-48

n77(config-if-range)#
• If you DON’T, traffic arriving on ports with default policy (i.e., without all queues
activated that have COS/DSCP values mapped) will suffer – packet drops, poor
performance, etc.
• Of course, you can have different non-default policies on different sets of interfaces,
but all interfaces in the system must use some policy that defines all activated queues!
Shape the SP Queue
(Optional) Clone a default egress “type

queuing” policy-map
Create a copy of a default egress queuing policy
Shape SP queue in new (cloned) “type

queuing” policy
Limit SP queue bandwidth consumption
Apply new “type queuing” policy to target

interface(s)
Apply new queuing policy to interfaces
Important: applying new queuing policy takes effect immediately and is disruptive to any
ports to which the policy is applied
Shape the SP Queue
Example: Shape the SP queue to 2Gbps on a 10G interface, using a queuing policy cloned from the default “8e4q4q”
egress queuing policy
n77# ! Clone the 8E egress queuing policy

n77# qos copy policy-map type queuing default-8e-4q8q-out-policy prefix new- Clone the default egress
n77# conf queuing policy
n77(config)# ! Modify new queuing policy
n77(config)# policy-map type queuing new-8e-4q8q-out
Modify the cloned policy
n77(config-pmap-que)# ! Modify egress queue q1
n77(config-pmap-que)# class type queuing 8e-4q8q-out-q1
n77(config-pmap-c-que)# ! Make this queue strict priority
Make q1 Strict Priority and
n77(config-pmap-c-que)# priority level 1
shape to 20% (2G)
n77(config-pmap-c-que)# ! Shape the queue to 20% (2G on 10G port)
n77(config-pmap-c-que)# shape percent 20
n77(config-pmap-c-que)# int e 2/1-48 Apply new policy to target
n77(config-if-range)# ! Apply the new policy to target interfaces interfaces
n77(config-if-range)# service-policy type queuing output new-8e-4q8q-out
n77(config-if-range)#
n77#
Changing The Default Trust
(M-Series I/O Module)
• You can make an interface untrusted (CoS and DSCP)
• CoS for bridged traffic policy-map type queuing Reset-CoS
• DSCP for routed traffic class type queuing 8q2t-in-q-default
set cos 0
• You need two Policies bandwidth percent 100
• A "type queuing" policy !
policy-map type qos Reset-DSCP
to set the CoS to 0 class class-default
• A "type qos" policy set dscp 0
!
to set the DSCP to 0 ! Tie to an interface:
service-policy type queuing input Reset-CoS
service-policy type qos input Reset-DSCP
 Set DSCP will set the CoS
value for Bridged traffic as well.
Changing The Default Trust
(F-Series I/O Module)
qos copy policy-map type queuing default-4q-8e-in-policy prefix UNTRUSTED-

!
policy-map type queuing untrusted-4q-8e-in
class type queuing 2q4t-8e-in-q1
class type queuing 2q4t-8e-in-q-default
set cos 0
!
policy-map type qos UNTRUSTED
class class-default
set dscp 0
!
! Tie to an interface:
service-policy type queuing input untrusted-4q-8e-in
service-policy type qos input UNTRUSTED
Complete your
online session • Please complete your session survey
evaluation after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live
Mobile App or by logging in to the Session
Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing
on demand after the event at ciscolive.cisco.com.
Continue your education
Demos in the
Walk-in labs
Cisco campus
Meet the engineer

Related sessions
1:1 meetings
Thank you
#CLUS
#CLUS

BRKDCN 3346

Uploaded by

Copyright:

Available Formats

BRKDCN 3346

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 3346

Uploaded by

Copyright:

Available Formats

#CLUS

Webex Teams will be moderated cs.co/ciscolivebot#BRKDCN-3346

Maximise Throughput and Manage Congestion!

Identify and Discard Mark Traffic Prioritise, Control

• Conform Action (permit)

• Two rate Three Color Policer

Ingress Module Egress Module

Ingress Module Egress Module

Ingress Module Egress Module

Ingress Module Egress Module

Ingress Module Egress Module

Ingress Module Egress Module

Ingress Egress Ingress Egress

N ports N ports N ports N ports

• Round Robin (RR) • Deficit Weighted Round Robin

• Weighted Round Robin (WRR) • Shaped Round Robin

• Assumes a mean packet size

• Weighted Random Early Drop (WRED)

What Traffic do we care about? Policy-Map

Class-map Policy-map Service-policy

• One network-QoS policy per system, applies to all ports

• Assumption is network-QoS policy defined/applied consistently network-wide

Network QoS policies should be applied consistently

Switch 1 Switch 2 Switch 3

Ingress Ingress Ingress

Nexus(config)# system qos

configured in VLAN Database Access Port

Nexus(config)# vlan configuration <vlan-id>

takes precedence over VLAN applied to the

channel and applies to all

member-ports Routed Port Port-channel

• No Egress QoS policies on L2 … or a port-

Nexus(config)# interface ethernet 1/1

takes precedence over VLAN applied to the

configured under the

respective Interface Routed Port Port-Channel

• Queuing Policy can be … or a port-

attached to port-channel also channel

Nexus(config)# interface ethernet 1/1

• ECN (Explicit Congestion Notification)

• Uses two LSB bits in DiffServ field

• PFC Applications Applications Applications

• ETS FCP RDMA API RDMA API

Requirement for RoCEv2

• DiffServ Tunneling mode provides 3 3 24 3

EXP 3 EXP 3 EXP 3

DSCP 24 DSCP 24 DSCP 24 DSCP 24 DSCP 24

CE1 PE1 P1 P2 PE2 CE2

EXP 3 EXP 3 EXP 2

DSCP 24 DSCP 24 DSCP 24 DSCP 24 DSCP 16

CE1 PE1 P1 P2 PE2 CE2

EXP 3 EXP 3 EXP 5

DSCP 24 DSCP 24 DSCP 24 DSCP 24 DSCP 24

CE1 PE1 P1 P2 PE2 CE2

VXLAN Encap. Packet 5 46

DSCP 26 DSCP 26 VXLAN EVPN DSCP 26 DSCP 26

VTEP VTEP VTEP VTEP VTEP

COS 3 DSCP 26 COS 0 DSCP 26

Host A Host C Host B