Cisco Nvme Fundamental

#CLMEL
Deep Dive into NVMe™ and NVMe™ over Fabrics

(NVMe-oF™)
BRKDCN-2494
J Metz, Ph.D, R&D Engineer, Advanced Storage

Board of Directors, NVM Express, Inc.
Board of Directors, SNIA
Board of Directors, FCIA
@drjmetz
#CLMEL
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Open the Cisco Events Mobile App
2 Find your desired session in the “Session Scheduler”
3 Click “Join the Discussion”
4 Install Webex Teams or go directly to the team space
5 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKDCN-2494
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction
• Storage Concepts
• What is NVMe™
• NVMe Operations
• The Anatomy of NVM Subsystems
• Queuing and Queue Pairs
• NVMe™ over Fabrics (NVMe-oF™)

• How NVMe-oF Works
• NVMe-oF™/RDMA
• NVMe-oF™/FC
• NVMe-oF™/TCP
• Whats New in NVMe 1.3
• Additional Resources
#CLMEL
BRKDCN-2494 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
What This Presentation Is… and Is Not
• What it is:
• A technology conversation
• Deep dive (We’re going in, Jim!)
• What it is not
• A product conversation
• Comprehensive and exhaustive
• A discussion of best practices (see, e.g.,
BRKDCN-2729 Networking Impact of
NVMe over Fabrics)
#CLMEL BRKDCN-2494 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Goals
At the end of this presentation you should know:

• What NVMe is and why it is important
• How NVMe is extended for remote access over
a network (i.e., “Fabrics”)
• The different types of fabrics and their
differences
• Some of the differences between traditional
SCSI-based storage solutions and NVMe-
based solutions
• What’s new in NVMe and NVMe over Fabrics
#CLMEL
Prerequisites
• You really should know…

• Basics of Block-based storage
• Basic terminology (initiator, target)
• Helpful to know…
• Basic PCIe semantics
• Some storage networking
Note: Screenshot Warning!
Get your screenshots ready when you see this symbol:
Useful for getting more information about a topic, URLs, etc.
What We Will Not Cover (In Detail)
• NVMe-MI (Management)
• NVMe-KV (Key Value)
• Protocol Data Protection features
• Advances in NVMe features
• RDMA verbs
• Fibre Channel exchanges
• New form factor designs
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Who is NVM Express?
About NVM Express (The Technology)
• NVM Express (NVMe™) is an open collection of standards and information to fully expose the benefits of non-
volatile memory in all types of computing environments from mobile to data center
• NVMe™ is designed from the ground up to deliver high bandwidth and low latency storage access for current and
future NVM technologies
NVM Express Base Specification

The register interface and command set for PCI Express attached storage with industry standard software available for numerous operating
systems. NVMe™ is widely considered the defacto industry standard for PCIe SSDs.
NVM Express Management Interface (NVMe-MI™) Specification

The command set and architecture for out of band management of NVM Express storage (i.e., discovering, monitoring, and updating NVMe™
devices using a BMC).
NVM Express Over Fabrics (NVMe-oF™) Specification

The extension to NVM Express that enables tunneling the NVM Express command set over additional transports beyond PCIe. NVMe over
Fabrics™ extends the benefits of efficient storage architecture at scale in the world’s largest data centers by allowing the same protocol to
extend over various networked interfaces.
BRKDCN-2494
The Basics of Storage
and Memory
The Anatomy of Storage
• There is a “sweet spot” for storage
• Depends on the workload and application type
• No “one-size fits all”
• Understanding “where” the solution fits is critical
to understanding “how” to put it together
• Trade-offs between 3 specific forces
• “You get, at best, 2 out of 3”
The Anatomy of Storage
• There is a “sweet spot” for storage
• Depends on the workload and application type
• No “one-size fits all”
• Understanding “where” the solution fits is critical to
understanding “how” to put it together
• Trade-offs between 3 specific forces
• “You get, at best, 2 out of 3”
• NVMe goes here
Storage Solutions - Where Does It Fit?
• Different types of storage apply in different
places
Storage Solutions - Where Does NVMe Fit?
places
• NVMe is PCIe-based
• Local storage, in-server
Storage Solutions - Where Does NVMe Fit?
places
• PCIe Extensions (HBAs and switches) extend NVMe
outside the server
Storage Solutions - Where Does NVMe-oF Fit?
places
• PCIe Extensions (HBAs and switches) extend NVMe
outside the server
• NVMe-oF inherits characteristics of

underlying transport
But First… SCSI
• SCSI is the command set used in traditional storage
• Is the basis for most storage used in the data center
• Obviously, is the most obvious starting point for working with Flash storage
• These commands are transported via:
• Fibre Channel
• Infiniband
• iSCSI (duh!)
• SAS, SATA
• Works great for data that can’t be accessed in parallel (like disk drives that rotate)
• Any latency in protocol acknowledgement is far less than rotational head seek time
Evolution from Disk Drives to SSDs
The Flash Conundrum
• Flash
• Requires far fewer commands than SCSI provides
• Does not rotate (no latency time, exposing latency
of a one command/one queue system)
• Thrives on random (non-linear) access
• Both read and write
• Nearly all Flash storage systems historically use
SCSI for access
• But they don’t have to!
So, NVMe…
• Specification for SSD access via PCI Express (PCIe), initially flash media
• Designed to scale to any type of Non Volatile Memory, including Storage Class Memory
• Design target: high parallelism and low latency SSD access
• Does not rely on SCSI (SAS/FC) or ATA (SATA) interfaces: New host drivers & I/O stacks
• Common interface for Enterprise & Client drives/systems: Reuse & leverage engineering investments
• New modern command set
• 64-byte commands (vs. typical 16 bytes for SCSI)
• Administrative vs. I/O command separation (control path vs. data path)
• Small set of commands: Small fast host and storage implementations
• Standards development by NVMe working group.
• NVMe is de facto future solution for NAND and post NAND SSD from all SSD suppliers
• Full support for NVMe for all major OS (Linux, Windows, ESX etc.)
• Learn more at nvmexpress.org
What is NVMe?
NVMe Operations
What’s Special about NVMe?
Slower Faster
HDD: varying speed conveyer belts Flash: all data blocks available at the Flash: all data blocks available at the
carrying data blocks (faster belts = same seek time & latency same seek time & latency
lower seek time & latency)
SCSI / SAS: pick & place robot SCSI /SAS: pick & place robot NVMe / PCIe: pick & place robot with
w/tracked, single arm executing 1 w/single arm executing 1 command at 1000s of arms, all processing &
command at a time, 1 queue a time, 1 queue executing commands simultaneously,
with high depth of commands
BRKDCN-2494 25
Technical Basics
• 2 key components: Host and NVM
Subsystem (a.k.a. storage target)
• “Host” is best thought of as the CPU and
NVMe I/O driver for our purposes
• “NVM Subsystem” has a component - the
NVMe Controller - which does the
communication work
Technical Basics
• Memory-based deep queues
(up to 64K commands per
queue, up to 64K queues)
• Streamlined and simple
command set (13; only 3
required commands)
• Command completion interface
optimized for success
(common case)
• NVMe Controller: SSD element
that processes NVMe
commands
#CLMEL BRKDCN-2494 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
The Anatomy of NVM
Subsystems
What You Need - NVM Subsystem Fabric Fabric
Port Port
• Architectural Elements
NVMe NVMe
• Fabric Ports controller controller
• NVMe Controllers NSID NSID NSID NSID
• NVMe Namespaces
• Implementation Dependent NVMe NVMe

Elements Namespace Namespace
• NVM Media and Interface
NVM Interface (I/F)
NVM Media NVM Media NVM Media
NVMe subsystem
Fabric Ports Fabric Fabric
Port Port
• Subsystem Ports are associated with

Physical Fabric Ports NVMe NVMe
controller controller
• In PCIe it’s a PCIe command
NSID NSID NSID NSID
• Multiple NVMe Controllers may be accessed
through a single port
• NVMe Controllers are associated with one NVMe NVMe
port Namespace Namespace
• Fabric Types; PCIe, RDMA (Ethernet
RoCE/iWARP, InfiniBand™), Fibre Channel, NVM Interface (I/F)
TCP NVM Media NVM Media NVM Media
NVMe subsystem
NVMe Controller Fabric Fabric
Port Port
• NVMe Command Processing

• On PCIe systems, it’s associated with a single NVMe NVMe
PCIe function controller controller
• Access to NVMe Namespaces NSID NSID NSID NSID
• Namespace ID (NSID) associates a Controller to
Namespaces(s)
• May have multiple Controllers per NVM Subsystem
NVMe NVMe
• Used in multi-host and multi-path configurations
Namespace Namespace
• NVMe Queue Host Interface
• Paired Command Submission and Completion
Queues NVM Interface (I/F)
• Admin Queue for configuration, Scalable number NVM Media NVM Media NVM Media
of IO Queues
NVMe subsystem
NVMe Namespaces and NVM Media Fabric Fabric
Port Port
• Defines the mapping of NVM Media to

a formatted LBA range NVMe
controller
NVMe
controller
• Multiple formats supported with/without
end-to-end protection NSID NSID NSID NSID
• NVM Subsystem may have multiple

Namespaces
NVMe NVMe
• Private or Shared Namespaces Namespace Namespace
• Private is accessible by one Controller,
Shared accessible by multiple NVM Interface (I/F)
Controllers NVM Media NVM Media NVM Media
• Namespace Reservations
NVMe subsystem
NVMe Subsystem Implementations
NVMe PCIe SSD Implementation
(single Subsystem/Controller)
NVMe all NVM Storage Appliance Implementation

All NVM Appliance with
(1000’s of Subsystems/Controllers) PCIe NVMe SSDs
Queuing and Queue Pairs
In This Section…
• NVMe Host/Controller Communications
• Command Submission and Completion
• NVMe Multi-Queue Model Multi-queue
communication
• Command Data Transfers
NVMe Command
• NVMe communications over multiple fabric Submission and
Completion
transports
NVMe Host/Controller Communications
• NVMe Multi-Queue Interface Model Host

• Single Administrative and Multiple IO
Queues
• Host sends NVMe Commands over
the Submission Queue (SQ)
• Controller sends NVMe Completions
over a paired Completion Queue (CQ)
• Transport type-dependent interfaces
facilitate the queue operations and Transport-Dependent Interfaces
NVMe Command Data transfers
Memory PCIe Registers Fabric Capsule Operations
NVMe Multi-Queue Interface
• I/O Submission and Completion Host

Queue Pairs are aligned to Host NVMe Host Driver
CPU Cores
CPU Core 0 CPU Core 1…
• Independent per-queue
operations
• No inter-CPU locks on command
Submission or Completion
• Per Completion Queue Interrupts
enables source core interrupt Transport-Dependent Interfaces
steering
Memory PCIe Registers Fabric Capsule Operations
NVMe controller
Queues Scale With Controllers
Host
• Each Host/Controller pair have an NVMe Host Driver
independent set of NVMe queues CPU Core 0 CPU Core 1 CPU Core N-1
• Controllers and queues operate

autonomously
• NVMe Controllers may be local PCIe or
remote Fabric
• Use a common NVMe Queuing Model
NVMe controller NVMe controller
NVMe Commands and Completions
• NVMe Commands are sent by the Host to the
Controller in Submission Queue Entries (SQE)
• Separate Admin and IO Commands
• Three mandatory IO Commands
• Added two fabric-only Commands
• Commands may complete out of order
• NVMe Completions are sent by the Controller to
the Host in Completion Queue Entries (CQE)
• Command Id identifies the completed
command
• SQ Head Ptr indicates the consumed SQE
slots that are available for posting new SQEs
NVMe Generic Queuing Operational Model
Host
1. Host Driver enqueues the SQE NVMe Host Driver
into the SQ
Fabric
Port
NVMe controller
Host
into the SQ
2. NVMe Controller dequeues SQE
Fabric
Port
NVMe controller
Host
into the SQ
(A) Data transfer, if applicable, goes here
Fabric
Port
NVMe controller
Host
1. Host Driver enqueues the SQE into NVMe Host Driver
the SQ
3. NVMe Controller enqueues CQE
into the CQ
Fabric
Port
NVMe controller
Host
1. Host Driver enqueues the SQE into NVMe Host Driver
the SQ
into the CQ
4. Host Driver dequeues CQE
Fabric
Port
NVMe controller
Host
into the SQ
into the CQ
4. Host Driver dequeues CQE
Fabric
This queuing functionality is always present… Port
… but where this takes place can differ NVMe controller
NVMe Queuing on Memory (PCIe)
Host
1.Host Driver enqueues the SQE in NVMe Host Driver
host-memory resident SQ Enqueue

SQE
Memory Transport (PCIe)

Fabric
Port
NVMe controller
Host

SQE
2.Host Driver notifies controller

about new SQE by writing doorbell
register 2
PCIe Write
(SQ Doorbell)

Fabric
Port
NVMe controller
Host

SQE

about new SQE by writing doorbell 3
register 2
PCIe Read
Response
PCIe Read (SQE)
(SQE)
3.NVMe Controller dequeues SQE

PCIe Write
(SQ Doorbell)

by reading it from the host Fabric
Port
memory SQ
Dequeue
SQE
NVMe controller
Host

SQE

about new SQE by writing 3
doorbell register 2
PCIe Read
Response
PCIe Read (SQE)
(SQE)
3.NVMe Controller dequeues SQE

PCIe Write
(SQ Doorbell)

by reading it from the host Fabric
Port
memory SQ
Dequeue
(A) Data transfer, if applicable, goes SQE
here NVMe controller
Host

SQE
2.Host Driver notifies controller about

1
new SQE by writing doorbell register

3
3.NVMe Controller dequeues SQE by 2
PCIe Read
reading it from the host memory SQ

Response
PCIe Read (SQE)
(SQE)
PCIe Write 4
(SQ Doorbell)

PCIe Write
4.NVMe Controller enqueues CQE by

(CQE)
Fabric
writing it to host-resident CQ Port
Dequeue
SQE
Enqueue
CQE
NVMe controller
Host
1.Host Driver enqueues the SQE in host- NVMe Host Driver
memory resident SQ Enqueue

SQE
Dequeue
CQE

1
5

3.NVMe Controller dequeues SQE by
3
2
reading it from the host memory SQ

PCIe Read
Response
PCIe Read (SQE)
(SQE)
PCIe Write 4
(SQ Doorbell)

PCIe Write
4.NVMe Controller enqueues CQE by (CQE)
writing it to host-resident CQ Fabric

Port
5.Host Driver dequeues CQE Dequeue

SQE
Enqueue
CQE
NVMe controller
Host

SQE
Dequeue
CQE

1
5

3.NVMe Controller dequeues SQE by
3
2
PCIe Read
reading it from the host memory SQ PCIe Write

PCIe Read
(SQE)
Response
(SQE)
4
(SQ Doorbell)

PCIe Write
4.NVMe Controller enqueues CQE by

(CQE)
writing it to host-resident CQ
Fabric
Port
5.Host Driver dequeues CQE Dequeue

SQE
Enqueue
CQE
NVMe controller
What is NVMe™ over
Fabrics (NVMe-oF™)?
What is NVMe over Fabrics (NVMe-oF™)?
• Built on common NVMe architecture with additional
definitions to support message-based NVMe
operations
• Expansion of NVMe semantics to remote storage
• Retains NVMe efficiency and performance over
network fabrics
• Eliminates unnecessary protocol translations
• Standardization of NVMe over a range Fabric types
• RDMA (RoCE, iWARP, InfiniBand™), Fibre Channel,
and TCP
What’s Special About NVMe over Fabrics?
Maintaining Consistency
• Recall: Host
NVMe Host Driver
• Multi-queue model
CPU Core 0 CPU Core n…
• Multipathing capabilities built-in
• Optimized NVMe System
• Architecture is the same,
regardless of transport
• Extends efficiencies across Transport-Dependent Interfaces
fabric Memory PCIe Registers Fabric Capsule Operations
NVMe controller
NVMe Multi-Queue Scaling
Host
• Queue pairs scale NVMe Host Driver
CPU Core 0 CPU Core 1 CPU Core N-1

• Maintain consistency to multiple
Subsystems
• Each controller provides a separate set of
queues, versus other models where Transport-Dependent Interfaces
single set of queues is used for multiple Fabric Capsule Operations
controllers
• Efficiency remains
NVMe controller NVMe controller
NVMe and NVMe-oF Models
• NVMe is a Memory-Mapped, PCIe Model
• Fabrics is message-based, shared memory is optional
• In-capsule data transfer is always message-based
NVMe Transports
Memory Message Message & Memory

Data & Commands/Responses
use shared memory Commands/Responses use Capsules Commands/Responses use Capsules
Data may use Capsules or Messages Data may use Capsules or Shared Memory
Example: Examples: Examples:

PCI Express Fibre Channel, RDMA
TCP (InfiniBand, RoCE, iWARP)
Fabric Message-Based Transports

Capsule = Encapsulated NVMe Command/Completion within a transport message
Data = Transport data exchange mechanism (if any)
What’s Special About NVMe-oF: Bindings
• What is a Binding?
• “A specification of reliable delivery of data,
commands, and responses between a host and an
NVM subsystem for an NVMe Transport. The
binding may exclude or restrict functionality based
on the NVMe Transport’s capabilities”
• I.e., it’s the “glue” that links all the pieces above
and below (examples):
• SGL Descriptions
• Data placement restrictions
• Data transport capabilities
• Authentication capabilities
Key Differences Between NVMe and NVMe-oF
• One-to-one mapping between I/O Submission Queues and I/O Completion Queues
• NVMe-oF does not support multiple I/O Submission Queues being mapped to a single I/O Completion Queue
• A controller is associated with only one host at a time

• NVMe over Fabrics allows multiple hosts to connect to different controllers in the NVM subsystem through the same port
• NVMe over Fabrics does not define an interrupt mechanism that allows a controller to generate a host interrupt
• It is the responsibility of the host fabric interface (e.g., Host Bus Adapter) to generate host interrupts
• NVMe over Fabrics Queues are created using the Connect Fabrics command
• Replaces PCIe queue creation commands
• If metadata is supported, it may only be transferred as a contiguous part of the logical block
• NVMe over Fabrics does not support transferring metadata from a separate buffer
• NVMe over Fabrics does not support PRPs but requires use of SGLs for Admin, I/O, and Fabrics commands
• This differs from NVMe over PCIe where SGLs are not supported for Admin commands and are optional for I/O commands
• NVMe over Fabrics does not support Completion Queue flow control
• This requires that the host ensures there are available Completion Queue slots before submitting new commands
NVMe over Fabrics Capsules
NVMe Fabric CMD Capsule
• NVMe over Fabric Command Capsule Command Id
• Encapsulated NVMe SQE Entry OpCode
NSID
• May contain additional Scatter Gather Lists (SGL) or NVMe
Command Data Buffer Optional Additional SGL(s)
Address
• Transport-agnostic Capsule format (PRP/SGL)
or
Command Data
Command
Parameters
• NVMe over Fabric Response Capsule

• Encapsulated NVMe CQE Entry NVMe Fabric CMD Capsule
• May contain NVMe Command Data Command Parm
• Transport-agnostic Capsule format SQ Head Ptr Optional
Command Status P Command Data
Command Id
NVMe-oF Encapsulation Process
NVMe NVMe-oF NVMe/RDMA RDMA QP
NVMe-oF
NVMe SQE NVMe-oF
Capsule
Capsule
Buffer Handle SGL Keyed SGL
NVMe/TCP
NVMe/TCP TCP/IP Byte Stream
Header
PCIe NVMe-oF
Capsule
NVMe SQE
Transport SGL
PRP/SGL
NVMe/FC FC Exchange
NVMe SQ FCP/NVMe
Header
FRAME FRAME FRAME
NVMe-oF
Capsule
Transport SGL
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62 62
NVMe-oF Queuing Interface to Transports
Host
1.Host Driver encapsulates SQE into 1
NVMe Host Driver
an NVMe-oF Command Capsule Encapsulate

SQE
Capsule Transport
NVMe controller
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Host
1.Host Driver encapsulates SQE into NVMe Host Driver
an NVMe-oF Command Capsule

2.NVMe-oF capsule is sent to the
Transport-Dependent Message
2
network/Fabric Send
Capsule
Capsule Transport
NVMe controller
Host

network/Fabric
3.Fabric enqueues the SQE into the
remote NVMe SQ Capsule Transport
Receive
Capsule Enqueue
SQE
NVMe controller
Host
1.Host Driver encapsulates SQE into 1
NVMe Host Driver
an NVMe-oF Command Capsule Encapsulate

SQE
2
network/Fabric Send
Capsule

(A)Data transfer, if applicable, goes here
Receive
Capsule Enqueue
SQE
NVMe controller
Host

network/Fabric
4.Controller encapsulates CQE into an

NVMe-oF Response Capsule
Encapsulate
CQE
4
NVMe controller
Host
1. Host Driver encapsulates SQE into an NVMe Host Driver
NVMe-oF Command Capsule

2. NVMe-oF capsule is sent to the
network/Fabric
3. Fabric enqueues the SQE into the remote
NVMe SQ
Send
Capsule
4. Controller encapsulates CQE into an NVMe- Capsule Transport

oF Response Capsule
5. NVMe-oF Response capsule is sent to the
network/Fabric
4
NVMe controller
Host
1. Host Driver encapsulates SQE into an NVMe- NVMe Host Driver
6
oF Command Capsule Decapsulate

CQE

network/Fabric
3. Fabric enqueues the SQE into the remote
NVMe SQ
4. Controller encapsulates CQE into an NVMe-oF
Response Capsule Capsule Transport

network/Fabric
6. Fabric enqueues the CQE into the host CQ
NVMe controller
Host
1. Host Driver encapsulates SQE into an NVMe- 1
NVMe Host Driver
6
oF Command Capsule Encapsulate

SQE
Decapsulate
CQE

network/Fabric
2
3. Fabric enqueues the SQE into the remote Send

Capsule
NVMe SQ Send
Capsule
4. Controller encapsulates CQE into an NVMe-oF 5
Response Capsule Capsule Transport

network/Fabric Receive
Capsule Enqueue
SQE
6. Fabric enqueues the CQE into the host CQ Encapsulate

CQE
3
4
NVMe controller
NVMe Data Transfers
NVMe Command Data Transfers
Command Id
OpCode Physical Region Page (PRP)
NSID Physical Memory Page Address
Used by Memory Transport (PCIe)
Buffer
or
Address
(PRP/SGL)
Scatter Gather List (SGL)

Command Used by Memory and Capsule Transports
Parameters
Physical, logical key, or capsule offset address
• SQE contains the NVMe Command Data buffer address

• Physical Region Page (PRP) used only for PCIe Transport
• Scatter Gather List used by both PCIe and Capsule Transports
• SGL = [Address, Length]
• Address may be physical, logical with key, or capsule offset based
• Supports SGL lists; { [Address,Length]...[Address,Length] }
NVMe-oF Data Transfers (Memory + Messages)
Host
• Command and Response Capsules are Host Memory
NVMe Host Driver
transferred using messages Buffer
• Data is transferred using memory

semantics
• Data transfer operations are transport
Transport
specific; examples
• RDMA: RDMA_READ/RDMA_WRITE
Operations
• Similar to PCIe (PCIe Memory Read/Write
Requests)
Read Command Data Write Command Data
NVMe controller
NVM Subsystem 7373

NVMe-oF Data Transfers (Messages Only)
Host
• Command Capsules, Response NVMe Host Driver
Host Memory
Buffer
Capsules transferred using
messages
• Data is transferred using messages
Transport
• Data transfer operations are
transport specific; examples
• Fibre Channel: FCP Exchanges
• TCP: H2C and C2H PDUs
Read Command Data Write Command Data
NVMe controller
NVM Subsystem 7474

RDMA-based NVMe-oF
What is Remote Direct Memory Access (RDMA)?
• RDMA is a host-offload, host-bypass technology that allows
an application (including storage) to make data transfers
directly to/from another application’s memory space
• The RDMA-capable Ethernet NICs (RNICs) – not the host –
rNIC/HCA rNIC/HCA
manage reliable connections between source and
destination
• Applications communicate with the RDMA NIC using
dedicated Queue Pairs (QPs) and Completion Queues (CQs)
• Each application can have many QPs and CQs
• Each QP has a Send Queue (SQ) and Receive Queue (RQ)
• Each CQ can be associated with multiple SQs or RQs
Benefits of RDMA User
S/W
User Application
IO Library
• Bypass of system software stack components that
processes network traffic Kernel Apps
• For user applications (outer rails), RDMA bypasses the Kernel OS Stack
kernel altogether S/W
Sys Driver
• For kernel applications (inner rails), RDMA bypasses
the OS stack and the system drivers
PCIe
• Direct data placement of data from one machine (real or H/W
virtual) to another machine – without copies Transport &
Network (L4/ L3)
• Increased bandwidth while lowering latency, jitter, and
CPU utilization Ethernet (L1/ L0)
• Great for networked storage!

Standard RDMA
NIC Flow NIC Flow
Queues, Capsules, and More Queues
Example of Host Write To Remote Target
• NVMe Host Driver encapsulates the NVMe Submission Queue Entry (including
data) into a fabric-neutral Command Capsule and passes it to the NVMe RDMA
Transport
• Capsules are placed in Host RNIC RDMA Send Queue and become an
RDMA_SEND payload
• Target RNIC at a Fabric Port receives Capsule in an RDMA Receive Queue
• RNIC places the Capsule SQE and data into target host memory
• RNIC signals the RDMA Receive Completion to the target’s NVMe RDMA
Transport
• Target processes NVMe Command and Data
• Target encapsulates the NVMe Completion Entry into a fabric-neutral Response
Capsule and passes it to NVMe RDMA Transport
Source: SNIA
#CLMEL
NVMe Multi-Queue Host Interface Map to RDMA
Queue-Pair
Standard (local) NVMe
Model
• NVMe Submission and Completion
Queues are aligned to CPU cores
• No inter-CPU software locks
• Per CQ MSI-X interrupts enable source
core interrupt steering
NVMe Over RDMA Fabric
• Retains NVMe SQ/CQ CPU alignment

• No inter-CPU software locks
• Source core interrupt steering retained
by using RDMA Event Queue MSI-X
interrupts
Varieties of RDMA-based NVMe-oF
• RoCE is based on InfiniBand transport over Ethernet
• RoCEv2
• Enhances RoCE with a UDP header (not TCP) and
Internet routability
• Uses InfiniBand transport on top of Ethernet
• iWARP is layered on top of TCP/IP
• Offloaded TCP/IP flow control and management
• Both iWARP and RoCE (and InfiniBand) support verbs
• NVMe-oF using Verbs can run on top of either
transport
NVMe over Fabrics and RDMA
• InfiniBand
• RoCE v1 (generally
not used)
• RoCE v2 (most
popular vendor
option)
• iWARP
Compatibility Considerations
Choose one
• iWARP and RoCE are software-compatible if written
to the RDMA Verbs
• iWARP and RoCE both require rNICs
• iWARP and RoCE cannot talk RDMA to each other
because of L3/L4 differences
• iWARP adapters can talk RDMA only to iWARP
adapters
• RoCE adapters can talk RDMA only to RoCE
adapters
No mix and match!
#CLMEL
Key Takeaways
Things to remember about RDMA-based NVMe-oF
• NVMe-oF requires the low network latency that RDMA can provide
• RDMA reduces latency, improves CPU utilization
• NVMe-oF supports RDMA verbs transparently
• No changes to applications required
• NVMe-oF maps NVMe queues to RDMA queue pairs
• RoCE and iWARP are software compatible (via Verbs) but do not
interoperate because their transports are different
• RoCE and iWARP
• Different vendors and ecosystem
• Different network infrastructure requirements
FC-NVMe: Fibre Channel-
based NVMe-oF
Fibre Channel Protocol
Fibre Channel has layers, just like
Upper Layer
• FC-4 Protocol Interface
OSI and TCP

FC-3 Common Services
• At the top level is the Fibre Channel
Protocol (FCP)
FC-2 Framing and Flow Control
• Integrates with upper layer protocols, such
as SCSI, FICON, and NVMe
FC-1 Byte Encoding
FC-0 Physical Interface
What Is FCP?
• What’s the difference between FCP and
“FCP”?
• FCP is a data transfer protocol that carries
other upper-level transport protocols
(e.g., FICON, SCSI, NVMe)
• Historically FCP meant SCSI FCP, but
other protocols exist
• NVMe “hooks” into FCP
• Seamless transport of NVMe traffic
• Allows high performance HBA’s to work
with FC-NVMe
FCP Mapping
• The NVMe Command/Response capsules, and for some commands,
data transfer, are directly mapped into FCP Information Units (IUs)
• A NVMe I/O operation is directly mapped to a Fibre Channel Exchange
FC-NVMe Information Units (IUs)
#CLMEL
Zero Copy
• Zero-copy
• Allows data to be sent to user application with minimal
copies
• RDMA is a semantic which encourages more efficient
data handling, but you don’t need it to get efficiency
• FC has had zero-copy years before there was RDMA
• Data is DMA’d straight from HBA to buffers passed to
user
• Difference between RDMA and FC is the APIs
• RDMA does a lot more to enforce a zero-copy
mechanism, but it is not required to use RDMA to get
zero-copy
FCP Transactions NVMe Initiator NVMe Target
• NVMe-oF using Fibre FCP FCP
Channel Transactions look I/F Model I/F Model
similar to RDMA
• For Read
• FCP_DATA from Target
IO Read IO Write
• For Write Initiator Target Initiator Target
• Transfer Ready and then DATA to

Target
RDMA Transactions NVMe-oF NVMe-oF Target
Initiator
• NVMe-oF over RDMA NVMe-oF NVMe-oF
protocol transactions I/F Model I/F Model
• RDMA Write
• RDMA Read with RDMA Read
Response
IO Read IO Write
Initiator Target Initiator Target
FC-NVMe: More Than The Protocol
• Dedicated Storage Network
• Run NVMe and SCSI Side-by-Side
• Robust and battle-hardened
discovery and name service
• Zoning and Security
• Integrated Qualification and Support
TCP-based NVMe-oF
NVMe-TCP NVMe™ Host Software
• NVMe™ block storage protocol over standard Host Side Transport Abstraction
TCP/IP transport
• Enables disaggregation of NVMe™ SSDs without
Next Gen Fabrics

compromising latency and without requiring
Fibre Channel
changes to networking infrastructure
RDMA
TCP
• Independently scale storage & compute to
maximize resource utilization and optimize for
specific workload requirements
• Maintains NVMe™ model: sub-systems, controllers
namespaces, admin queues, data queues Controller Side Transport Abstraction
9
BRKDCN-2494
4
Why NVMe™/TCP?
• Ubiquitous - runs on everything everywhere…
• Well understood - TCP is probably the most common transport
• High performance - TCP delivers excellent performance scalability
• Well suited for large scale deployments and longer distances
• Actively developed - maintenance and enhancements are developed by major players
• Inherently supports in-transit encryption
• The Network Infrastructure may not be well designed to meet Latency

requirements
• Some Application may have strict performance thresholds than vanilla TCP can
provide
NVMe/TCP PDUs and Capsules Segmentation
NVMe/TCP Controller Association
NVMe-oF Host
Controller association maps 1x1 NVMe queue to a TCP

Admin I/O I/O I/O I/O
•
connection
• No controller-wide sequencing
• No controller-wide reassembly constraints
• No shared state across NVMe queues and TCP
connections accessed in the “hot” path
• Each NVMe queue (and its backing TCP connection)
can be assigned to a separate CPU core.
• Connection binding is performed in NVMe-oF connect

time (binding queue to controller)
Admin I/O I/O I/O I/O
NVMe-oF Controller
#CLMEL © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 97
NVMe-TCP Data Path Usage
• Enables NVMe-oF I/O operations in
existing IP Datacenter environments
• Software-only NVMe Host Driver
with NVMe-TCP transport
• (Eventually) Can provide an NVMe-oF alternative to iSCSI for

Storage Systems with PCIe NVMe SSDs
• More efficient End-to-End NVMe Operations by eliminating
SCSI to NVMe translations
• Co-exists with other NVMe-oF transports
• Transport selection may be based on h/w support and/or policy
Source: Dave Minturn (Intel)

NVMe-TCP Control Path Usage
• Enables use of NVMe-oF on Control-
Path Networks (example: 1g Ethernet)
• Discovery Service Usage
• Discovery controllers residing on a
common control network that is
separate from data-path networks
• NVMe-MI Usage
• NVMe-MI endpoints on control
processors (BMC, ..) with simple IP
network stacks (1g Ethernet)
• NVMe-MI on separate control

network
Source: Dave Minturn (Intel)

NVMe-oF Key Takeaways
• NVMe over Fabrics – expansion of NVMe to allow scaling of ecosystem to larger disaggregated systems
• Majority of capacity is shipped in external storage
• Keep same semantics as NVMe
• NVMe-oF technology is still in early stages of deployment
• Primarily supported in latest Linux distros
• Proper sizing – optimize deployment to use case
• Customers with RDMA experience will lean towards NVMe over RDMA
• Customers with large FC infrastructure may find transition to NVMe over FC easier
• Multiple deployment options
• “Front end” or “Back end” connectivity, disaggregated/top of rack flash deployment
What’s New in NVMe
1.3
Source: NVM Express, Inc.
What’s New in NVMe 1.3

Type Description Benefit
Boot Partitions Enables bootstrapping of an SSD in a low resource environment
Client/Mobile Host Controlled
Host control to better regulate system thermals and device throttling
Thermal Management
Enables exchange of meta data between device and host. First use is
Directives
Streams to increase SSD endurance and performance
Provides more flexibility with shared storage use cases and resource
Data Center/Enterprise Virtualization assignment, enabling developers to flexibly assign SSD resources to
specific virtual machines
Emulated Controller
Better performance for software defined NVMe controllers
Optimization
Timestamp Start a timer and record time from host to controller via set and get features
Debug Error Log Updates Error logging and debug, root cause problems faster
Telemetry Standard command to drop telemetry data, logs
Device Self-Test Internal check of SSD health, ensure devices are operating as expected
Simple, fast, native way to completely erase data in an SSD, allowing more
Sanitize
Management options for secure SSD reuse or decommissioning
Management
Allows same management commands in or out-of-band
Enhancements
SGL Dword
Storage Simpler implementation
Simplification
#CLMEL
NVMe Status and Feature Roadmap
2014 2015 2016 2017 2018 2019
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
NVM Express
NVMe 1.2 Nov’14 NVMe 1.2.1 May’16 NVMe

NVMe 1.3 1.3
May’17 NVMe 1.4
(NVMe)
• Namespace Management • Sanitize • IO Determinism

• Controller Memory Buffer • Streams • Persistent Memory Region
• Host Memory Buffer • Virtualization • Persistent Event Log
• Live Firmware Update • Multipathing
NVMe Over
(NVMe-oF)
NVMe-oF 1.0 May’16 NVMe-oF 1.1

Fabrics
• Transport and protocol • Enhanced Discovery

• RDMA binding • In-band Authentication
• TCP Transport Binding
Interface (NVMe-
NVMe Mgmt.
NVMe-MI 1.0a NVMe-MI 1.1

NVMe-MI 1.0 Nov’15
April’17
• Enclosure Management
MI)
• Out-of-band management
• Device discovery • In-band Mechanism
• Health & temp monitoring • Storage Device Extension
• Firmware Update
Released NVMe specification Planned release

#CLMEL
Additional Resources
• NVMe For Absolute Beginners
• https://blogs.cisco.com/datacenter/nvme-for-absolute-beginners
• NVMe over Fabrics for Absolute Beginners
• https://jmetz.com/2018/08/nvme-over-fabrics-for-absolute-beginners/
• NVMe Program of Study
• https://jmetz.com/2016/08/learning-nvme-a-program-of-study/
• A NVMe Bibliography
• https://jmetz.com/2016/02/a-nvme-bibliography/
• NVMe and Fibre Channel
• https://jmetz.com/2016/09/fibre-channel-and-nvme-over-fabrics/
• Latest Developments in NVMe/TCP
• https://www.snia.org/sites/default/files/SDC/2018/presentations/NVMe/Grimberg_Sagi_Latest_De
velopments_with_NVMe_TCP.pdf
• Other Storage Topics, Webinars, Articles, Podcasts
• jmetz.com/bibliography
Summary
or, How I Learned to Stop Worrying and Love the Key Takeaways
Summary
• NVMe and NVMe-oF
• Treats storage like memory, just with permanence
• Built from the ground up to support a consistent model for NVM interfaces, even
across network fabrics
• No translation to or from another protocol like SCSI (in firmware/software)
• Inherent parallelism of NVMe multiple I/O Queues is exposed to the host
• NVMe commands and structures are transferred end-to-end, and architecture is
maintained across a range of fabric types
Q&A
#CLMEL
Complete Your Online Session Evaluation
• Give us your feedback and receive a
complimentary Cisco Live 2019
Power Bank after completing the
overall event evaluation and 5 session
evaluations.
• All evaluations can be completed via
the Cisco Live Melbourne Mobile App.
• Don’t forget: Cisco Live sessions will
be available for viewing on demand
after the event at:
https://ciscolive.cisco.com/on-demand-library/
Thank you
#CLMEL
#CLMEL

Cisco Nvme Fundamental

Uploaded by

Copyright:

Available Formats

Cisco Nvme Fundamental

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco Nvme Fundamental

Uploaded by

Copyright:

Available Formats

What is NVMe and why is it important?

What is NVMe and why is it important?

How is NVMe extended for remote access over a network?

How is NVMe extended for remote access over a network?

#CLMEL

Deep Dive into NVMe™ and NVMe™ over Fabrics

J Metz, Ph.D, R&D Engineer, Advanced Storage

• NVMe™ over Fabrics (NVMe-oF™)

At the end of this presentation you should know:

• You really should know…

Get your screenshots ready when you see this symbol:

Useful for getting more information about a topic, URLs, etc.

NVM Express Base Specification

NVM Express Management Interface (NVMe-MI™) Specification

NVM Express Over Fabrics (NVMe-oF™) Specification

• NVMe-oF inherits characteristics of

• Implementation Dependent NVMe NVMe

NVM Media NVM Media NVM Media

• Subsystem Ports are associated with

TCP NVM Media NVM Media NVM Media

• NVMe Command Processing

• Defines the mapping of NVM Media to

• NVM Subsystem may have multiple

Controllers NVM Media NVM Media NVM Media

NVMe all NVM Storage Appliance Implementation

• NVMe Multi-Queue Interface Model Host

• I/O Submission and Completion Host

• Controllers and queues operate

NVMe controller NVMe controller

… but where this takes place can differ NVMe controller

host-memory resident SQ Enqueue

Memory Transport (PCIe)

host-memory resident SQ Enqueue

2.Host Driver notifies controller

Memory Transport (PCIe)

host-memory resident SQ Enqueue

2.Host Driver notifies controller

3.NVMe Controller dequeues SQE

Memory Transport (PCIe)

host-memory resident SQ Enqueue

2.Host Driver notifies controller

3.NVMe Controller dequeues SQE

Memory Transport (PCIe)

(A) Data transfer, if applicable, goes SQE

here NVMe controller

host-memory resident SQ Enqueue

2.Host Driver notifies controller about

new SQE by writing doorbell register

reading it from the host memory SQ

Memory Transport (PCIe)

4.NVMe Controller enqueues CQE by

memory resident SQ Enqueue

2.Host Driver notifies controller about

new SQE by writing doorbell register

reading it from the host memory SQ

Memory Transport (PCIe)

4.NVMe Controller enqueues CQE by (CQE)

writing it to host-resident CQ Fabric

5.Host Driver dequeues CQE Dequeue