RDMA Over Converged Ethernet
RDMA Over Converged Ethernet
RDMA Over Converged Ethernet
Outline
InfiniBand pros
InfiniBand cons
RoCE
InfiniBand vs RoCE
What should you be doing
InfiniBand pros
Well understood widely deployed HPC
interconnect
Approximately 50% of Top 500 list
Characteristics:
High bandwidth
Low latency
Strong reliability
Rich set of messaging protocols
Layered architecture (OSI model based)
Green low CPU utilization and power
InfiniBand cons
Its not a LAN
Not routable (at least in production)
Scaling limitations (approx. 10K nodes)
Not ubiquitous (cant connect multiple clusters in
one big fabric)
Its not Ethernet (some folks just wont make the
leap)
RoCE
Stands for RDMA over Converged Ethernet is
an IBTA standard.
InfiniBand L3-Ln over Ethernet Layer L1-L2
Effectively InfiniBand and Ethernet have
converged their phys at 40G and above; so really
just InfiniBand with an Ethernet L2