What is InfiniBand network and the difference with Ethernet?

What is InfiniBand network?


The InfiniBand architecture brings fabric consolidation to the data center Storage networking can concurrently run with clustering, communication and management fabrics over the same infrastructure, preserving the behavior of multiple fabrics.  

InfiniBand is an open standard network interconnection technology with high bandwidth, low delay and high reliability. This technology is defined by IBTA (InfiniBand trade alliance). This technology is widely used in the field of supercomputer cluster. At the same time, with the rise of artificial intelligence, it is also the preferred network interconnection technology for GPU server.

The InfiniBand standard supports single data rate (SDR) signaling at a basic rate of 2.5Gbits/sec per lane to allow raw data rate of 10Gbits/sec over 4X cables (the most common InfiniBand cable type used). Double data rate (DDR) and quad data rate (QDR) signaling permit single lanes to be scaled up to 5Gbits/sec and 10Gbits/sec per lane, respectively, for a potential maximum data rate of 40Gbits/sec over 4X and 120Gbits/sec over 12X cables.

Comparison of networking technologies

Comparison of networking technologies

At present, the latest InfiniBand product is the HDR produced by mellanox, which can provide end-to-end bandwidth of up to 200gbps for the network, bring unparalleled network experience for high-performance computing, artificial intelligence and other fields, and maximize the computing potential in the cluster.

Whats 200G InfiniBand HDR?


InfiniBand supports SDR/DDR/QDR/FDR/EDR transmission to increase link bandwidth. Recently, mellanox released 200G InfiniBand supporting HDR. Mellanox recently released 200G InfiniBand with HDR support. Mellanox 200Gb/s HDR InfiniBand networks support ultra-low latency, high data throughput and intelligent network compute acceleration engines. Users can use standard Mellanox software drivers on the cloud, just as they would in a Bare Metal environment. With support for RDMA verbs, all InfiniBand-based MPI software such as Mellanox HPC-X, MVAPICH2, Platform MPI, Intel MPI, and more can be used.

In addition, users can also take advantage of the hardware offload feature of MPI cluster communication for additional performance gains, which also improves the efficiency of business applications. 200G InfiniBand has a wide range of applications, including in-network computing acceleration engines, HDR InfiniBand adapters, HDR InfiniBand Quantum switches and 200G cabling.

InfiniBand applications

InfiniBand applications

As for the 200G InfiniBand cabling, the final piece of the Mellanox 200Gbs solution is its line of LinkX cables. Mellanox offers direct-attach 200G copper cables reaching up to 3 meters and 2 x 100G splitter breakout cables to enable HDR100 links, as well as 200G active optical cables that reach up to 100 meters. All LinkX cables in the 200Gb/s line come in standard QSFP56 packages. 

What are the advantages of InfiniBand network?


  • Serial High Bandwidth Links

 – SDR: 10Gb/s

– DDR: 20Gb/s

– QDR: 40Gb/s

– FDR: 56Gb/s

– EDR: 100Gb/s

– HDR: 200Gb/s

  • Ultra-low latency

– Under 1 us application to application

  • Reliable, lossless, self-managing fabric

– Link level flow control

– Congestion control to prevent HOL blocking

  • Full CPU Offload

– Hardware-Based Reliable Transport Protocol

– Kernel Bypass (User level applications get direct access to hardware)

  • Memory exposed to remote node access – RDMA-read and RDMA-write

– Atomic operations

  • Quality Of Service

– Independent I/O channels at the adapter level

– Virtual Lanes at the link level

  • Cluster Scalability/flexibility

– Up to 48K nodes in subnet, up to 2^128 in network

– Parallel routes between end nodes

– Multiple cluster topologies possible

  • Simplified Cluster Management

– Centralized route manager

– In-band diagnostics and upgrades

What is Ethernet network?


Ethernet refers to the baseband LAN specification standard created by Xerox company and jointly developed by Xerox, Intel and DEC company. The general Ethernet standard was issued on September 30, 1980. It is the most general communication protocol standard adopted by the existing LAN. It transmits and receives data through cables. Ethernet network is used to create local area network and connect multiple computers or other devices such as printers, scanners, and so on. In a wired network, this is done with the help of fiber optic cables, while in a wireless network, it is done through wireless network technology. The main types of Ethernet networks are Fast Ethernet, Gigabit Ethernet, 10-Gigabit Ethernet and Switch Ethernet.

At present, the IEEE 802.3 Standard Organization organized by IEEE has issued Ethernet interface standards of 100GE, 200GE and 400GE. Ethernet network is the transmission technology with the highest rate at present.

InfiniBand vs Ethernet: whats the difference?


As interconnection technologies, InfiniBand and Ethernet have their own characteristics and differences. They are developing and evolving in their different application fields, and have become two indispensable Interconnection Technologies in our network world.

Types of Ethernet network

Types of Ethernet network

In terms of Ethernet network, along with the IP technology, they constitute the cornerstone of the whole Internet building in the world. All people and intelligent devices rely on Ethernet to realize the interconnection of all things, which is related to the original intention of its design to achieve better compatibility. It can make different systems better interconnected, which makes Ethernet have very strong adaptability since its birth. After decades of development, it has become the standard of the Internet.  

In terms of InfiniBand network, it is an interconnection standard to solve the data transmission bottleneck in high-performance computing scenarios. It has been positioned as a high-end application since its formulation. Interconnection is not the main contradiction, and high-performance communication is the main entry point. Infiniband is a newer type of connection that was recently released. The great thing about it is the speed that users are provided with. While the speed for your connection will ultimately depend on what wire you choose, the speeds for them can go up to 40Gbps or evermore.

Therefore, compared with Ethernet technology, InfiniBand is inherently different from Ethernet due to different positioning, mainly in bandwidth, delay, network reliability and networking mode. InfiniBand directly creates a private and protected channel between nodes through switches to transmit data and messages without CPU participating in remote direct memory access (RDMA). One end of the adapter is connected to the CPU through the PCI Express interface, and the other end is connected to the InfiniBand subnet through the InfiniBand network port. Compared with other network communication protocols, this provides obvious advantages, including higher bandwidth, lower latency and enhanced scalability.



Leave a Comment

Scroll to Top