IBTA Launches XDR (800G) InfiniBand Specification

IBTA launches XDR InfiniBand specification, providing unprecedented 800Gb/s server and storage connectivity for the next generation of artificial intelligence and scientific computing, and further enhancing the capabilities of AI and HPC infrastructure.

Source: InfiniBand Roadmap – Advancing InfiniBand (infinibandta.org)

The InfiniBand roadmap is the result of the continuous improvement of multiple working groups within IBTA, whose members include leading enterprise IT vendors who actively contribute to the development of InfiniBand technology. The roadmap details the development direction of 1x, 2x, 4x, and 12x port widths. Its main goal is to ensure that the improvement of InfiniBand performance is consistent with the growth of system-level performance.

The InfiniBand Trade Association (IBTA) is a global organization dedicated to developing and further adopting RDMA technology. It recently announced the launch of IBTA Specification Volume 1 Release 1.7 and Volume 2 Release 1.5. Volume 1 Release 1.7 includes the initial specification of XDR and lays the foundation for the next generation of speeds in the IBTA roadmap, providing twice the bandwidth of the previous generation to meet the high-speed performance and scalability requirements of AI and HPC data centers.

SerDes supports 200Gb/s per channel, and XDR-enabled network cards and switches will provide unprecedented 800Gb/s per port speed, and support 1.6Tb/s XDR switch-to-switch connections.

Other features in Volume 1 Release 1.7 and Volume 2 Release 1.5 specifications include improved support for large-scale high-order switches; improved congestion control mechanisms in InfiniBand and RoCE architectures using network probes, and the final physical specification for NDR 400Gb/s InfiniBand.

“High-bandwidth, low-latency and energy-efficient interconnects are essential for artificial intelligence and HPC applications that involve large-scale data processing and analysis, such as scientific simulations, deep learning, and training neural networks on massive data sets.” “XDR InfiniBand lays the foundation for the next generation of artificial intelligence and scientific computing with unprecedented 800Gb/s speed, enhances the capabilities of artificial intelligence applications, and further advances the frontiers of knowledge and discovery.”

Volume 1 of the InfiniBand specification includes the core of the InfiniBand architecture, specifying the requirements for InfiniBand switches, routers, and adapters. It also specifies the complete architecture for managing InfiniBand structures. Its companion document Volume 2 specifies the physical aspects of the architecture and more.

The main updates of Volume 1 Release 1.7 are:

Specification update overview

Support for large radix switches

Update DR for large radix switches

Next generation speed

An introduction to the network probe issue:

Network probes are a generic mechanism for probing the network state.

Probes are sent from one endpoint to another and may interact with network entities along the way.

Network probes can be used to collect information about the network without requiring specific processes to run on remote nodes.

Network Probes use the basic MAD format and appear as standard MAD packets in the network.

RDMA

Network probing design guidelines

Network probing architecture