{"id":1437,"date":"2024-06-21T17:12:00","date_gmt":"2021-12-01T08:51:08","guid":{"rendered":"https:\/\/www.fibermall.com\/blog\/?p=1437"},"modified":"2026-03-13T09:17:14","modified_gmt":"2026-03-13T09:17:14","slug":"what-is-infiniband-network-and-the-difference-with-ethernet","status":"publish","type":"post","link":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm","title":{"rendered":"What is InfiniBand Network and the Difference with Ethernet?"},"content":{"rendered":"\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#History_and_Standards\" >History and Standards<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#What_is_the_InfiniBand_Network\" >What is the InfiniBand Network?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#InfiniBand_Overview\" >InfiniBand Overview<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Whats_200G_InfiniBand_HDR\" >What&#8217;s 200G InfiniBand HDR?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#What_are_the_Advantages_of_InfiniBand_Network\" >What are the Advantages of InfiniBand Network?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#What_is_an_Ethernet_Network\" >What is an Ethernet Network?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#InfiniBand_vs_Ethernet_Whats_the_Difference\" >InfiniBand vs Ethernet: What&#8217;s the Difference?<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Types_of_Ethernet_Networks\" >Types of Ethernet Networks<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Technical_Comparison_of_InfiniBand_and_Ethernet\" >Technical Comparison of InfiniBand and Ethernet<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Performance_and_Scalability\" >Performance and Scalability<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#InfiniBand_vs_Omni-Path_Advantages_of_InfiniBand_over_Omni-Path\" >InfiniBand vs Omni-Path: Advantages of InfiniBand over Omni-Path<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#InfiniBand_HDR_Product_Introduction\" >InfiniBand HDR Product Introduction<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#HDR_Switch\" >HDR Switch<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#HDR_Network_Adapter\" >HDR Network Adapter<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#InfiniBand_Packets_and_Data_Transfer\" >InfiniBand Packets and Data Transfer<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#InfiniBand_Layered_Architecture\" >InfiniBand Layered Architecture<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#The_Switching_Mechanism_of_InfiniBand\" >The Switching Mechanism of InfiniBand<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Cost_and_Operational_Considerations\" >Cost and Operational Considerations<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#InfiniBand_vs_Ethernet_Cost\" >InfiniBand vs. Ethernet Cost<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-20\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Maintenance_and_Operating_Expenses\" >Maintenance and Operating Expenses<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-21\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Power_Consumption_and_Efficiency_Gains\" >Power Consumption and Efficiency Gains<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-22\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Choosing_the_Right_Technology\" >Choosing the Right Technology<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-23\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Best_Ways_to_Optimize_Infrastructure_for_AI_Workloads\" >Best Ways to Optimize Infrastructure for AI Workloads<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-24\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#AI_Infrastructure_Cloud_vs_Edge_vs_On-Premises\" >AI Infrastructure: Cloud vs. Edge vs. On-Premises<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-25\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Choosing_the_Right_Technology_for_Your_AI_Networking_Needs\" >Choosing the Right Technology for Your AI Networking Needs<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-26\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Frequently_Asked_Questions\" >Frequently Asked Questions<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-27\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Is_Ethernet_Better_Than_InfiniBand\" >Is Ethernet Better Than InfiniBand?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-28\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Is_InfiniBand_Copper_or_Fiber\" >Is InfiniBand Copper or Fiber?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-29\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Is_InfiniBand_Owned_by_NVIDIA\" >Is InfiniBand Owned by NVIDIA?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-30\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#How_Fast_Is_InfiniBand\" >How Fast Is InfiniBand?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-31\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Why_Is_InfiniBand_So_Fast\" >Why Is InfiniBand So Fast?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-32\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Who_Uses_InfiniBand\" >Who Uses InfiniBand?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-33\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#How_Do_I_Optimize_My_Network_for_AI_Workloads\" >How Do I Optimize My Network for AI Workloads?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-34\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\/#Related_Posts\" >Related Posts<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\" id=\"h-history-and-standards\"><span class=\"ez-toc-section\" id=\"History_and_Standards\"><\/span><strong>History and Standards<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>InfiniBand, developed in the late 1990s, was designed for high-speed, low-latency interconnects in HPC environments. It operates under standards set by the InfiniBand Trade Association (IBTA), with modern versions like HDR and NDR supporting speeds up to 400 Gbps. Ethernet, dating back to the 1970s, is governed by the IEEE 802.3 standards and has evolved from 10 Mbps to 400 Gbps and beyond, making it a flexible choice for diverse applications. Understanding their historical context and standards helps clarify their roles in modern networking.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-is-the-infiniband-network\"><span class=\"ez-toc-section\" id=\"What_is_the_InfiniBand_Network\"><\/span><strong>What is the InfiniBand Network?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The InfiniBand architecture brings fabric consolidation to the data center Storage networking can concurrently run&nbsp;with clustering, communication and management fabrics over the same infrastructure, preserving the behavior of multiple fabrics.<\/p>\n\n\n\n<p>InfiniBand is an open standard network interconnection technology with high bandwidth, low delay and high reliability. This technology is defined by IBTA&nbsp;(InfiniBand trade alliance). This technology is widely used in the field of supercomputer cluster. At the same time, with the rise of artificial intelligence, it is also the preferred network interconnection technology for GPU server.<\/p>\n\n\n\n<p>High Speed Interconnection (HSI) has become the key to the development of high performance computers as the computing power of the Central Processing Unit (CPU) is increasing at a very fast pace. HSI is a new technology proposed to improve the performance of the Peripheral Component Interface (PCI). After years of development, HSIs supporting High Performance Computing (HPC) are now mainly Gigabit Ethernet and <a href=\"https:\/\/www.fibermall.com\/store-21993-linkx-transceivers-and-cables.htm\" target=\"_blank\" rel=\"noreferrer noopener\">InfiniBand<\/a>, of which InfiniBand is the fastest growing HSI. InfiniBand is a high-performance, low-latency technology developed under the supervision of the InfiniBand Trade Association (IBTA).<\/p>\n\n\n\n<p>IBTA was founded in 1999 as a merger of two industry organizations, Future I\/O Developers Forum and NGI\/O Forum.&nbsp;It works under the leadership of a planning and operating committee consisting of HP, IBM, Intel, Mellanox, Oracle, QLogic, Dell, Bull, and others. IBTA specializes in product compliance and interoperability testing, and its members have been working to advance the establishment and updating of the InfiniBand specification.<\/p>\n\n\n\n<p>The InfiniBand standard supports single data rate (SDR) signaling at a basic rate of 2.5Gbits\/sec per lane to allow raw data rate of 10Gbits\/sec over 4X cables (the most common InfiniBand cable type used). Double data rate (DDR) and quad data rate (QDR) signaling permit single lanes to be scaled up to 5Gbits\/sec and 10Gbits\/sec per lane, respectively, for a potential maximum data rate of 40Gbits\/sec over 4X and 120Gbits\/sec over 12X cables.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/Comparison-of-networking-technologies.png\" alt=\"Comparison of networking technologies\"\/><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><span style=\"font-size: 10pt\"><strong>Comparison of networking technologies<\/strong><\/span><\/p>\n\n\n\n<p>At present, the latest InfiniBand product is the HDR produced by Mellanox, which can provide end-to-end bandwidth of up to 200gbps for the network, bring unparalleled network experience for high-performance computing, artificial intelligence, and other fields, and maximize the computing potential in the cluster.<\/p>\n\n\n\n<p>As a computer cluster interconnection technology, InfiniBand has significant advantages over Ethernet\/Fibre Channel and the obsolete Omni-Path technology, and is the primary recommended network communication technology by the InfiniBand Trade Association (IBTA). Since 2014, most of the TOP500 supercomputers have adopted InfiniBand networking technology. In recent years, AI\/Big Data-related applications have also adopted IB networks on a large scale to achieve high-performance cluster deployments, with 62% of the Top100 supercomputing centers using InfiniBand technology (June 2022 data).<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/top100-infiniband.png\" alt=\"top100 infiniband\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-infiniband-overview\"><span class=\"ez-toc-section\" id=\"InfiniBand_Overview\"><\/span><strong>InfiniBand Overview <\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>InfiniBand is a communication link for data flow between processors and I\/O devices, supporting up to 64,000 addressable devices.&nbsp;InfiniBand Architecture (<a href=\"https:\/\/www.infinibandta.org\/\" target=\"_blank\" rel=\"nofollow noreferrer noopener\">IBA<\/a>) is an industry standard specification that defines a point-to-point switched input\/output framework for interconnecting servers, communications infrastructure, storage devices, and embedded systems.<\/p>\n\n\n\n<p>InfiniBand is ideal for connecting multiple data streams (clustering, communication, storage, management) in a single connection, with thousands of interconnected nodes, due to its pervasive, low latency, high bandwidth, and low management cost. The smallest complete IBA unit is a subnet, and multiple subnets are connected by routers to form a large IBA network.<\/p>\n\n\n\n<p>InfiniBand systems consist of channel adapters, switches, routers, cables, and connectors.&nbsp;The CA is divided into a Host Channel Adapter (HCA) and a Target Channel Adapter (TCA). IBA switches are similar in principle to other standard network switches, but must meet the high performance and low cost requirements of InfiniBand. An HCA is a device point through which an IB end node, such as a server or storage device, connects to an IB network.&nbsp;TCAs are a special form of channel adapters, mostly used in embedded environments such as storage devices.<\/p>\n\n\n\n<p>The InfiniBand architecture is shown in the figure.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1352\" height=\"874\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-architecture.png\" alt=\"infiniband architecture\" class=\"wp-image-4161\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-architecture.png 1352w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-architecture-300x194.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-architecture-1024x662.png 1024w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-architecture-768x496.png 768w\" sizes=\"(max-width: 1352px) 100vw, 1352px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-s-200g-infiniband-hdr\"><span class=\"ez-toc-section\" id=\"Whats_200G_InfiniBand_HDR\"><\/span><strong>What&#8217;s 200G InfiniBand HDR?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>InfiniBand supports SDR\/DDR\/QDR\/FDR\/EDR transmission to increase link bandwidth. Recently, Mellanox released 200G InfiniBand supporting HDR. Mellanox recently released <a href=\"https:\/\/www.fibermall.com\/store-21996-200g-hdr-infiniband.htm\" target=\"_blank\" rel=\"noreferrer noopener\">200G InfiniBand<\/a> with HDR support. Mellanox <a href=\"https:\/\/www.fibermall.com\/store-20654-200g-qsfp56.htm\" target=\"_blank\" rel=\"noopener\">200Gb\/s<\/a> HDR InfiniBand networks support ultra-low latency, high data throughput, and intelligent network compute acceleration engines. Users can use standard Mellanox software drivers on the cloud, just as they would in a Bare Metal environment. With support for RDMA verbs, all InfiniBand-based MPI software such as Mellanox HPC-X, MVAPICH2, Platform MPI, Intel MPI, and more can be used.<\/p>\n\n\n\n<p>In addition, users can also take advantage of the hardware offload feature of MPI cluster communication for additional performance gains, which also improves the efficiency of business applications. 200G InfiniBand has a wide range of applications, including in-network computing acceleration engines, HDR InfiniBand adapters, HDR InfiniBand Quantum switches and 200G cabling.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-applications.png\" alt=\"InfiniBand applications\"\/><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><span style=\"font-size: 10pt\"><strong>InfiniBand applications<\/strong><\/span><\/p>\n\n\n\n<p>As for the 200G InfiniBand cabling, the final piece of the Mellanox 200Gbs solution is its line of <a href=\"https:\/\/www.fibermall.com\/blog\/mellanox-infiniband-dac-aoc-cables.htm\" target=\"_blank\">LinkX cables<\/a>. Mellanox offers direct-attach 200G copper cables reaching up to 3 meters and 2 x 100G splitter breakout cables to enable HDR100 links, as well as 200G active optical cables that reach up to 100 meters. All LinkX cables in the 200Gb\/s line come in standard <a class=\"wpil_keyword_link\" href=\"https:\/\/www.fibermall.com\/blog\/qsfp56.htm\" title=\"QSFP56\" data-wpil-keyword-link=\"linked\" target=\"_blank\">QSFP56<\/a> packages.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"200G QSFP56 AOC: High-Speed Switch and NIC Connectivity Tests | FiberMall\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/VutJ-UFxkNg?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-are-the-advantages-of-infiniband-network\"><span class=\"ez-toc-section\" id=\"What_are_the_Advantages_of_InfiniBand_Network\"><\/span><strong>What are the Advantages of InfiniBand Network?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Serial High Bandwidth Links<\/li>\n<\/ul>\n\n\n\n<p>\u2013 SDR: 10Gb\/s<\/p>\n\n\n\n<p>\u2013 DDR: 20Gb\/s<\/p>\n\n\n\n<p>\u2013 QDR: 40Gb\/s<\/p>\n\n\n\n<p>\u2013 FDR: 56Gb\/s<\/p>\n\n\n\n<p>\u2013 EDR: 100Gb\/s<\/p>\n\n\n\n<p>\u2013 HDR: 200Gb\/s<\/p>\n\n\n\n<p>\u2013 <a href=\"https:\/\/www.fibermall.com\/store-21995-400g-ndr-infiniband.htm\" target=\"_blank\" rel=\"noreferrer noopener\">NDR: 400Gb\/s<\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ultra-low latency<\/li>\n<\/ul>\n\n\n\n<p>\u2013 Under 1 us application to application<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Reliable, lossless, self-managing fabric<\/li>\n<\/ul>\n\n\n\n<p>\u2013 Link level flow control<\/p>\n\n\n\n<p>\u2013 Congestion control to prevent HOL blocking<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Full CPU Offload<\/li>\n<\/ul>\n\n\n\n<p>\u2013 Hardware-Based Reliable Transport Protocol<\/p>\n\n\n\n<p>\u2013 Kernel Bypass (User level applications get direct access to hardware)<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Memory exposed to remote node access \u2013 RDMA-read and RDMA-write<\/li>\n<\/ul>\n\n\n\n<p>\u2013 Atomic operations<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Quality Of Service<\/li>\n<\/ul>\n\n\n\n<p>\u2013 Independent I\/O channels at the adapter level<\/p>\n\n\n\n<p>\u2013 Virtual Lanes at the link level<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Cluster Scalability\/flexibility<\/li>\n<\/ul>\n\n\n\n<p>\u2013 Up to 48K nodes in subnet, up to 2^128 in network<\/p>\n\n\n\n<p>\u2013 Parallel routes between end nodes<\/p>\n\n\n\n<p>\u2013 Multiple cluster topologies possible<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Simplified Cluster Management<\/li>\n<\/ul>\n\n\n\n<p>\u2013 Centralized route manager<\/p>\n\n\n\n<p>\u2013 In-band diagnostics and upgrades<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-is-an-ethernet-network\"><span class=\"ez-toc-section\" id=\"What_is_an_Ethernet_Network\"><\/span><strong>What is an Ethernet Network?<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Ethernet refers to the baseband LAN specification standard created by Xerox company and jointly developed by Xerox, Intel, and DEC company. The general Ethernet standard was issued on September 30, 1980. It is the most general communication protocol standard adopted by the existing LAN. It transmits and receives data through cables. <a href=\"https:\/\/www.fibermall.com\/store-21999-ethernet-transceivers-cables.htm\" target=\"_blank\" rel=\"noreferrer noopener\">Ethernet network<\/a> is used to create local area networks and connect multiple computers or other devices such as printers, scanners, and so on. In a wired network, this is done with the help of fiber optic cables, while in a wireless network, it is done through wireless network technology. The main types of Ethernet networks are Fast Ethernet, Gigabit Ethernet, 10-Gigabit Ethernet and Switch Ethernet.<\/p>\n\n\n\n<p>At present, the IEEE 802.3 Standard Organization organized by IEEE has issued Ethernet interface standards of 100GE, 200GE, and 400GE. Ethernet network is the transmission technology with the highest rate at present.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"400G InfiniBand\/Ethernet Flat Top OSFP SR4 Transceiver Demo | FiberMall\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/c7_0Ed0rWX0?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-infiniband-vs-ethernet-what-s-the-difference\"><span class=\"ez-toc-section\" id=\"InfiniBand_vs_Ethernet_Whats_the_Difference\"><\/span><a href=\"https:\/\/www.fibermall.com\/blog\/infiniband-vs-ethernet-in-hpc.htm\" target=\"_blank\" rel=\"noreferrer noopener\">InfiniBand vs Ethernet<\/a>: What&#8217;s the Difference?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>As interconnection technologies, InfiniBand and Ethernet have their own characteristics and differences. They are developing and evolving in their different application fields, and have become two indispensable Interconnection Technologies in our network world.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/types-of-Ethernet-network.png\" alt=\"Types of Ethernet network\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-types-of-ethernet-networks\"><span class=\"ez-toc-section\" id=\"Types_of_Ethernet_Networks\"><\/span><span style=\"font-size: 12pt\"><strong>Types of Ethernet Networks<\/strong><\/span><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>In terms of the Ethernet network, along with the IP technology, they constitute the cornerstone of the whole Internet building in the world. All people and intelligent devices rely on Ethernet to realize the interconnection of all things, which is related to the original intention of its design to achieve better compatibility. It can make different systems better interconnected, which makes Ethernet have very strong adaptability since its birth. After decades of development, it has become the standard of the Internet.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"What is InfiniBand Network and the Difference with Ethernet? | FiberMall\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/cowASe-dc7o?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<p>In terms of the InfiniBand network, it is an interconnection standard to solve the data transmission bottleneck in high-performance computing scenarios. It has been positioned as a high-end application since its formulation. Interconnection is not the main contradiction, and high-performance communication is the main entry point. Infiniband is a newer type of connection that was recently released. The great thing about it is the speed that users are provided with. While the speed for your connection will ultimately depend on what wire you choose, the speeds for them can go up to 40Gbps or evermore.<\/p>\n\n\n\n<p>Therefore, compared with Ethernet technology, <a href=\"https:\/\/www.infinibandta.org\/about-infiniband\/\" target=\"_blank\" rel=\"nofollow noopener\">InfiniBand<\/a> is inherently different from Ethernet due to different positioning, mainly in bandwidth, delay, network reliability and networking mode. InfiniBand directly creates a private and protected channel between nodes through switches to transmit data and messages without CPU participating in remote direct memory access (RDMA). One end of the adapter is connected to the CPU through the PCI Express interface, and the other end is connected to the InfiniBand subnet through the InfiniBand network port. Compared with other network communication protocols, this provides obvious advantages, including higher bandwidth, lower latency and enhanced scalability.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"800G InfiniBand Finned-Top OSFP SR8\/2xSR4 Transceiver | FiberMall\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/zomeamRU-8M?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-technical-comparison-of-infiniband-and-ethernet\"><span class=\"ez-toc-section\" id=\"Technical_Comparison_of_InfiniBand_and_Ethernet\"><\/span><strong>Technical Comparison of InfiniBand and Ethernet<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-basic-architecture-and-topology\"><strong>Basic Architecture and Topology<\/strong><strong><\/strong><\/h4>\n\n\n\n<p>InfiniBand uses a switched fabric architecture, enabling point-to-point&nbsp;connections with minimal latency. Its topology, often a fat-tree or torus, is optimized for HPC and AI clusters, supporting remote direct memory access (RDMA) for efficient data transfer. Ethernet, built on a layered architecture (e.g., TCP\/IP), supports diverse topologies like star or mesh, making it adaptable but potentially less efficient for AI-specific tasks. InfiniBand&#8217;s streamlined design gives it an edge in latency-sensitive environments, while Ethernet&#8217;s flexibility suits broader applications.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-key-technical-specifications\"><strong>Key Technical Specifications<\/strong><strong><\/strong><\/h4>\n\n\n\n<p>InfiniBand leverages protocols like RDMA over Converged Ethernet (RoCE) and supports both copper and fiber cabling, with high-bandwidth options like NDR (400 Gbps). Ethernet, with standards like 100GbE and 400GbE, supports similar cabling but relies on TCP\/IP or UDP for data transfer, which can introduce overhead. InfiniBand&#8217;s hardware, such as host channel adapters (HCAs), is optimized for specific workloads, while Ethernet&#8217;s switches and NICs are more generalized, impacting performance in niche scenarios.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-latency-and-speed-comparisons\"><strong>Latency and Speed Comparisons<\/strong><strong><\/strong><\/h4>\n\n\n\n<p>InfiniBand excels in latency, often achieving sub-microsecond delays, critical for AI training where GPUs or TPUs exchange massive datasets. Ethernet, while fast (e.g., 100GbE with ~1-2 \u00b5s latency), typically lags due to protocol overhead. Speed-wise, both technologies offer comparable headline rates (e.g., 400 Gbps), but InfiniBand&#8217;s RDMA and congestion management ensure more consistent performance under heavy AI workloads.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-bandwidth-and-data-transfer-rates\"><strong>Bandwidth and Data Transfer Rates<\/strong><strong><\/strong><\/h4>\n\n\n\n<p>InfiniBand&#8217;s high bandwidth (up to 400 Gbps per port in NDR) supports the massive data transfer needs of AI model training and large-scale simulations. Ethernet matches these rates in high-end configurations but may face bottlenecks in congested networks due to its reliance on TCP\/IP. For data centers handling petabytes of data, InfiniBand&#8217;s predictable throughput is a key advantage.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-scalability-in-expanding-networks\"><strong>Scalability in Expanding Networks<\/strong><strong><\/strong><\/h4>\n\n\n\n<p>InfiniBand scales efficiently in large clusters, with its fabric architecture minimizing latency as nodes increase. Its native support for partitioning and quality of service (QoS) ensures performance in complex AI environments. Ethernet scales well in enterprise settings but may require advanced configurations (e.g., spine-leaf architectures) to match InfiniBand&#8217;s efficiency in HPC or AI clusters.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-performance-and-scalability\"><span class=\"ez-toc-section\" id=\"Performance_and_Scalability\"><\/span><strong>Performance and Scalability<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>For AI workloads, InfiniBand&#8217;s low latency and high bandwidth provide superior performance, particularly for distributed training across GPU clusters. Ethernet performs well in general-purpose data centers but may struggle with the extreme demands of AI model synchronization. Both technologies scale, but InfiniBand&#8217;s design is better suited for massive, tightly coupled networks, while Ethernet excels in heterogeneous environments.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-application-scenarios-and-use-cases\"><strong>Application Scenarios and Use Cases<\/strong><strong><\/strong><\/h4>\n\n\n\n<p>InfiniBand dominates in data center and cloud deployments for AI, HPC, and big data analytics, powering systems like NVIDIA&#8217;s DGX platforms. It&#8217;s also prevalent in scientific and research institutions, supporting simulations in physics or genomics. Ethernet is preferred in enterprise data centers, cloud providers like AWS, and applications requiring broad compatibility, such as web hosting or storage networks. For AI-specific needs, InfiniBand&#8217;s performance edge makes it the go-to choice.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-infiniband-vs-omni-path-advantages-of-infiniband-over-omni-path\"><span class=\"ez-toc-section\" id=\"InfiniBand_vs_Omni-Path_Advantages_of_InfiniBand_over_Omni-Path\"><\/span><strong>InfiniBand vs Omni-Path: Advantages of InfiniBand over Omni-Path<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Although NVIDIA has launched the InfiniBand 400G NDR solution, some customers are still using 100G solution. For 100G high-performance networks, there are two common solutions, Omni-Path and InfiniBand, which have the same rate and similar performance, but the network structure is vastly different. For example, for a 400-node cluster, InfiniBand requires only 15 NVIDIA Quantum 8000 series switches and 200 200G branch cables, and 200 200G direct cables, while Omni-Path requires 24 switches and 876 100G direct cables (384 nodes). InfiniBand is very advantageous in the early equipment cost and later operation and maintenance cost, and the overall power consumption is much lower than Omni-Path, which is more environmentally friendly.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/Advantages-of-InfiniBand-over-Omni-Path.png\" alt=\"Advantages of InfiniBand over Omni-Path\"\/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-infiniband-hdr-product-introduction\"><span class=\"ez-toc-section\" id=\"InfiniBand_HDR_Product_Introduction\"><\/span><strong>InfiniBand HDR Product Introduction<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>EDR is being phased out of the market in terms of client demand, the NDR rate is too high and only head customers are trying to use it. HDR is widely used with the flexibility of HDR 100G and HDR 200G.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-hdr-switch\"><span class=\"ez-toc-section\" id=\"HDR_Switch\"><\/span>HDR Switch<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>There are two types of HDR switches. One is HDR CS8500. A 29U switch provides a maximum of 800 HDR 200Gb\/s ports, and each 200 GB port can be split into 2X100G to support 1600 HDR100 100Gb\/s ports.<\/p>\n\n\n\n<p>The other type is the QM8000 series. The 1U panel has 40 200G QSFP56 ports, which can be split into 80 HDR 100G ports at most for connecting 100G HDR network cards. At the same time, each port also supports EDR and directly connects with the network card of EDR. It should be noted that the 200G HDR port can only be slowed down to 100G and connected to the EDR network adapter, and cannot be split into 2X100G to connect two EDR network adapters.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/HDR-CS8500.png\" alt=\"HDR CS8500\"\/><\/figure>\n\n\n\n<p>There are two options for the 200G HDR switch: QM8700 and QM8790. The only difference between the two models is the management mode. The QM8700 provides the control interface for out-of-band management, while the QM8790 requires the NVIDIA Unified Fabric Manager (UFM\u00ae) platform for management.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/QM8000-series.png\" alt=\"QM8000 series\"\/><\/figure>\n\n\n\n<p>For QM8700 and QM8790, there are two airflow options for each model of the switch. One of them is 8790-HS2F for P2C airflow (front and rear airflow). The fan module is marked in blue. If you do not recognize the markings, you can also identify them by floating your hand over the air inlet and outlet of the switch.<\/p>\n\n\n\n<p>8790-HS2R is the red mark on the C2P airflow (rear front duct) fan module. Here P2C and C2P P means Power power, C means Cable (line interface), P2C (Power to Cable), C2P (Cable to Power) here the reference system is Power power side for the front, Cable line interface side for the rear.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/8790-HS2R-is-the-red-mark-on-the-C2P-airflow-rear-front-duct-fan-module..png\" alt=\"8790-HS2R is the red mark on the C2P airflow (rear front duct) fan module.\"\/><\/figure>\n\n\n\n<p>The QM8700 and QM8790 are generally used in two ways in practice, one is to interconnect with 200G HDR NICs by directly using 200G to <a href=\"https:\/\/www.fibermall.com\/sale-459098-mellanox-mfs1s00-h003e-200g-aoc-3m.htm\" target=\"_blank\" rel=\"noopener\"><span style=\"color: #0000ff\">200G AOC<\/span><\/a>\/DAC; the other common use is to interconnect with 100G HDR NICs by using 200G to 2X100G cables, in which one physical 200G (4X50G) QSFP56 port of the switch is split into 2 virtual 100G (2X50G) ports. 4X50G) QSFP56 port of the switch is split into two virtual 100G (2X50G) ports, and after the split, the symbol of the port is changed from x\/y to x\/Y\/z, where &#8220;x\/Y&#8221; indicates the previous symbol of the port before the split, and &#8220;z&#8221; denotes the number of the resulting single-lane port (1,2), and then each sub-physical port is treated as a single port.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/The-QM8700-and-QM8790-are-generally-used-in-two-ways-in-practice.png\" alt=\"The QM8700 and QM8790 are generally used in two ways in practice\"\/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-hdr-network-adapter\"><span class=\"ez-toc-section\" id=\"HDR_Network_Adapter\"><\/span>HDR Network Adapter<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>HDR network adapters are much more diverse than switches. The HDR100 network adapter supports a transmission rate of 100G. The two HDR100 ports can be connected to the HDR switch through 200G to 2x100G cables. Unlike the 100G EDR network card, the 100G port of the HDR100 network card supports both 4X25G NRZ and 2X50G PAM4 transmission. The HDR NIC supports a 200G transmission rate and can be connected to the switch through a 200G direct cable. In addition to the two interface rates, you can select single-port, dual-port, and PCIe network adapters of each rate based on service requirements. The common InfiniBand HDR network adapter models provided by FiberMall are as follows:<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/The-common-InfiniBand-HDR-network-adapter-models-provided-by-Fiber-Mall.png\" alt=\"The common InfiniBand HDR network adapter models provided by FiberMall\"\/><\/figure>\n\n\n\n<p>The InfiniBand network architecture is simple, but the solution&nbsp;selection is varied. 100G rate has both 100G EDR solution and 100G HDR solution; 200 rate also has HDR and 200G NDR two options. The network adapters, connectors, and switches used in different solutions are quite different.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-infiniband-packets-and-data-transfer\"><span class=\"ez-toc-section\" id=\"InfiniBand_Packets_and_Data_Transfer\"><\/span><b>InfiniBand Packets and Data Transfer<\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>A packet is the basic unit of InfiniBand data transmission. In order for information to propagate efficiently in an InfiniBand network, the information is split by the channel adapter into a number of packets. A complete IBA packet consists of Local Route Header, Global Route Header, Base Transport Header, Extended Transport Header, Payload&nbsp;(PYLD), Invariant CRC (ICRC) and Variant CRC (VCRC) fields, as shown in the figure below.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" width=\"812\" height=\"504\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-data-packet-format.png\" alt=\"infiniband data packet format\" class=\"wp-image-4162\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-data-packet-format.png 812w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-data-packet-format-300x186.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-data-packet-format-768x477.png 768w\" sizes=\"(max-width: 812px) 100vw, 812px\" \/><\/figure>\n\n\n\n<p><strong><b>LRH<\/b><\/strong>: 8 bytes, used by the switch to determine the local source and destination ports when forwarding packets and to regulate the class of service and virtual lane (VL) for packet transmission.<\/p>\n\n\n\n<p><strong><b>GRH<\/b><\/strong>: 40 bytes, used to route packets between subnets and ensure proper transmission of packets between subnets. It is specified by the Link Next Header (LNH) field in the LRH, using the IPv6 header specification defined in RFC 2460.<\/p>\n\n\n\n<p><strong><b>BTH<\/b><\/strong>: 12 bytes, specifying the destination Queue Pair (QP), indication opcode, packet serial number and segmentation.<\/p>\n\n\n\n<p><strong><b>ETH<\/b><\/strong>: 4-28 bytes, providing reliable datagram service.&nbsp;Payload (PYLD): 0-4096 bytes, the end-to-end application data being sent.<\/p>\n\n\n\n<p><strong><b>ICRC<\/b><\/strong>: 4 bytes, encapsulates the data that remains unchanged in the packet as it is sent from the source address to the destination address.<\/p>\n\n\n\n<p><strong><b>VCRC<\/b><\/strong>: 2 bytes, encapsulates the variable IBA and raw (raw) packets during the link.<\/p>\n\n\n\n<p>VCRC can be reconfigured in the fabric.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-infiniband-layered-architecture\"><span class=\"ez-toc-section\" id=\"InfiniBand_Layered_Architecture\"><\/span><strong>InfiniBand Layered Architecture <\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>According to the definition of IBTA, the InfiniBand architecture consists of physical layer, link layer, network layer and transport layer, and its layered architecture is shown in the figure.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter\"><img decoding=\"async\" width=\"972\" height=\"607\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-layered-architecture.png\" alt=\"infiniband layered architecture\" class=\"wp-image-4163\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-layered-architecture.png 972w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-layered-architecture-300x187.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/infiniband-layered-architecture-768x480.png 768w\" sizes=\"(max-width: 972px) 100vw, 972px\" \/><\/figure>\n\n\n\n<p><strong><b>Physical layer<\/b><\/strong>: The physical layer serves the link layer and provides the logical interface between these two layers. The physical layer consists of modules such as port signal connectors, physical connections (electrical and optical), hardware management, power management, and coding lines, whose main roles are to:<\/p>\n\n\n\n<p>(1) Establishing physical connections;<\/p>\n\n\n\n<p>(2) Notifying the link layer whether a physical connection is valid;<\/p>\n\n\n\n<p>(3) Monitoring the status of the physical connection, passing control signals and data to the link layer when the physical connection is valid, and transmitting the control and data information coming from the link layer.<\/p>\n\n\n\n<p><strong><b>Link layer<\/b><\/strong>: The link layer is responsible for handling the sending and receiving of link data in packets, providing services such as addressing, buffering, flow control, error detection and data exchange. Quality of service (QoS) is mainly reflected by this layer. The state machine is used to define the logical operations of the link layer as externally accessible operations, and does not specify internal operations.<\/p>\n\n\n\n<p><strong><b>Network layer<\/b><\/strong>: The network layer is responsible for routing packets between IBA subnets, including unicast and multicast operations. The network layer does not specify multiprotocol routing (e.g., IBA routing over non-IBA types), nor does it specify how the original packets are routed between IBA subnets.<\/p>\n\n\n\n<p><strong><b>Transport layer<\/b><\/strong>: Each IBA data contains a transport header. The transport header contains the information required by the end node to perform the specified operation. By manipulating QP, the IBA channel adapter communication clients at the transport layer form a &#8220;send&#8221; work queue and a &#8220;receive&#8221; work queue.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-the-switching-mechanism-of-infiniband\"><span class=\"ez-toc-section\" id=\"The_Switching_Mechanism_of_InfiniBand\"><\/span><b><strong>The Switching Mechanism of InfiniBand<\/strong><\/b><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The Switched Fabric used in InfiniBand is a switch-based point-to-point interconnection architecture geared toward system fault tolerance and scalability.<\/p>\n\n\n\n<p>IBA switches are the basic routing building blocks for internal subnet routing (inter-subnet routing functionality is provided by IBA routers). The interconnection of switches is accomplished by relaying packets between links.<\/p>\n\n\n\n<p>InfiniBand switches implement functions such as Subnet Manager Agent (SMA), Performance Manager Agent (PMA) and Baseboard Manager Agent (BMA). SMA provides an interface for subnet managers to obtain record and table data inside the switch via subnet management packets, implementing functions such as message notification, Service Level (SL) to Virtual Lane (VL) mapping, VL arbitration, multicast forwarding, and vendor characteristics. PMA provides an interface for performance managers to monitor performance information such as data throughput and error accumulation of the switch.BMA provides a communication channel between the baseboard manager and the bottom shelf manager.<\/p>\n\n\n\n<p>The main functions of data forwarding in InfiniBand switches are:<\/p>\n\n\n\n<p><strong><b>(1) Selecting the output port<\/b><\/strong>: Based on the Destination Local Identifier (DLID) of the packet, the switch finds out the port number of the output port from the forwarding table.<\/p>\n\n\n\n<p><strong><b>(2) Select output VL<\/b><\/strong>: SL and VL are supported, and the switch determines the VL of the output port used by packets of different priority levels based on the SL-VL mapping table.<\/p>\n\n\n\n<p><strong><b>(3) Data flow control<\/b><\/strong>: A credit-based link-level flow control mechanism is used.<\/p>\n\n\n\n<p><strong><b>(4) Support unicast, multicast and broadcast<\/b><\/strong>: The switch can convert multicast packets or broadcast packets into multiple unicast packets for exchange.<\/p>\n\n\n\n<p><strong><b>(5) Partitioning<\/b><\/strong>: Only hosts in the same partition can communicate with each other. Each partition has a unique partition key, and the switch checks whether the DLID of the packet is within the partition corresponding to the key.<\/p>\n\n\n\n<p><strong><b>(6) Error checking<\/b><\/strong>: including inconsistency error checking, encoding error checking, framing error checking, packet length checking, packet header version checking, service level validity checking, flow control compliance and maximum transmission unit checking.<\/p>\n\n\n\n<p><strong><b>(7) VL arbitration<\/b><\/strong>: Support subnet VL (including management VL15 and data VL). The switch uses VL arbitration to ensure that high-priority packets are better served.<\/p>\n\n\n\n<p>At present, the main manufacturers of InfiniBand switches are Mallanox, QLogic, Cisco, IBM, etc.<\/p>\n\n\n\n<p>For hosts, the client side of the transport layer is a Verbs software layer, where the client passes buffers or commands to and from these queues, and the hardware passes buffer data to and from them. When QP is established, it incorporates one of four IBA transport service types (reliable connection, reliable self-addressing information, unreliable self-addressing information, unreliable connection) or a non-IBA protocol encapsulated service. The transport service describes how the reliability and QP transport data works and what is transmitted.<\/p>\n\n\n\n<p>As an NVIDIA Elite level partner, FiberMall can provide complete <a href=\"https:\/\/www.fibermall.com\/collection\/infiniband.htm\" target=\"_blank\" rel=\"noopener\"><span style=\"color: #0000ff\">InfiniBand solutions<\/span><\/a> according to different customers&#8217; needs, and our senior technical engineers have rich experience in InfiniBand high-performance network solution design and project implementation services and can provide optimal solutions according to different application scenarios. We can provide QM8700\/QM8790 switch, HDR NIC, AOC\/DAC\/optical module portfolio solutions to achieve super performance and scalability, and improve ROI for HPC, AI, and other applications with lower cost and excellent performance.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-cost-and-operational-considerations\"><span class=\"ez-toc-section\" id=\"Cost_and_Operational_Considerations\"><\/span><strong>Cost and Operational Considerations<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-infiniband-vs-ethernet-cost\"><span class=\"ez-toc-section\" id=\"InfiniBand_vs_Ethernet_Cost\"><\/span><strong>InfiniBand vs. Ethernet Cost<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>InfiniBand&#8217;s initial investment is higher due to specialized hardware (e.g., HCAs, InfiniBand switches) and licensing costs. Ethernet benefits from widespread adoption, with lower-cost switches and NICs. However, for large-scale AI deployments, InfiniBand&#8217;s performance gains can offset upfront costs by reducing training times. Long-term costs depend on scale, with Ethernet often being more economical for smaller setups.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-maintenance-and-operating-expenses\"><span class=\"ez-toc-section\" id=\"Maintenance_and_Operating_Expenses\"><\/span><strong>Maintenance and Operating Expenses<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>InfiniBand requires specialized expertise for configuration and maintenance, increasing operational costs. Ethernet&#8217;s familiarity among IT staff reduces training needs, but complex Ethernet setups (e.g., for RDMA over Ethernet) may still demand expertise. Both require regular maintenance, such as cable inspections and firmware updates, but InfiniBand&#8217;s proprietary nature can complicate troubleshooting.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-power-consumption-and-efficiency-gains\"><span class=\"ez-toc-section\" id=\"Power_Consumption_and_Efficiency_Gains\"><\/span><strong>Power Consumption and Efficiency Gains<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>InfiniBand&#8217;s optimized design often results in lower power consumption per unit of data transferred, especially in high-throughput scenarios. Ethernet&#8217;s power usage varies by equipment, but high-end switches can consume more energy under heavy loads. For data centers prioritizing sustainability, InfiniBand&#8217;s efficiency is a compelling factor.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-choosing-the-right-technology\"><span class=\"ez-toc-section\" id=\"Choosing_the_Right_Technology\"><\/span><strong>Choosing the Right Technology<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-best-ways-to-optimize-infrastructure-for-ai-workloads\"><span class=\"ez-toc-section\" id=\"Best_Ways_to_Optimize_Infrastructure_for_AI_Workloads\"><\/span><strong>Best Ways to Optimize Infrastructure for AI Workloads<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>To optimize AI infrastructure, align the interconnect with workload demands. InfiniBand suits GPU-intensive tasks like deep learning, where low latency is critical. Ethernet is ideal for mixed workloads or environments requiring integration with existing networks. Key strategies include using RDMA (for both technologies), optimizing topology, and ensuring high-quality cabling.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-ai-infrastructure-cloud-vs-edge-vs-on-premises\"><span class=\"ez-toc-section\" id=\"AI_Infrastructure_Cloud_vs_Edge_vs_On-Premises\"><\/span><strong>AI Infrastructure: Cloud vs. Edge vs. On-Premises<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>In cloud environments, Ethernet dominates due to its standardization and compatibility with hyperscale providers. On-premises AI clusters often leverage InfiniBand for performance-critical tasks. Edge environments favor Ethernet for its flexibility and lower cost, though InfiniBand is emerging in edge HPC. Hybrid setups may combine both, using Ethernet for external connectivity and InfiniBand for internal clusters.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-choosing-the-right-technology-for-your-ai-networking-needs\"><span class=\"ez-toc-section\" id=\"Choosing_the_Right_Technology_for_Your_AI_Networking_Needs\"><\/span><strong>Choosing the Right Technology for Your AI Networking Needs<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Selecting between InfiniBand and Ethernet depends on:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Performance Needs: InfiniBand for low-latency, high-throughput AI tasks; Ethernet for general-purpose networking.<\/li>\n\n\n\n<li>Budget: Ethernet for cost-sensitive deployments; InfiniBand for performance-driven projects.<\/li>\n\n\n\n<li>Scale: InfiniBand for large AI clusters; Ethernet for smaller or mixed environments.<\/li>\n\n\n\n<li>Expertise: Ethernet if staff are familiar with TCP\/IP; InfiniBand if specialized skills are available.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-frequently-asked-questions\"><span class=\"ez-toc-section\" id=\"Frequently_Asked_Questions\"><\/span><strong>Frequently Asked Questions<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-is-ethernet-better-than-infiniband\"><span class=\"ez-toc-section\" id=\"Is_Ethernet_Better_Than_InfiniBand\"><\/span><strong>Is Ethernet Better Than InfiniBand?<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Ethernet is better for general-purpose, cost-sensitive, or broadly compatible networks. InfiniBand excels in AI, HPC, and latency-critical applications due to its performance advantages.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-is-infiniband-copper-or-fiber\"><span class=\"ez-toc-section\" id=\"Is_InfiniBand_Copper_or_Fiber\"><\/span><strong>Is InfiniBand Copper or Fiber?<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>InfiniBand supports both copper (for short distances, e.g., 5-10 meters) and fiber (for longer distances, up to kilometers), depending on the deployment.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-is-infiniband-owned-by-nvidia\"><span class=\"ez-toc-section\" id=\"Is_InfiniBand_Owned_by_NVIDIA\"><\/span><strong>Is InfiniBand Owned by NVIDIA?<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>InfiniBand is an open standard managed by the IBTA. NVIDIA develops InfiniBand hardware (e.g., ConnectX adapters) but does not own the technology.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-how-fast-is-infiniband\"><span class=\"ez-toc-section\" id=\"How_Fast_Is_InfiniBand\"><\/span><strong>How Fast Is InfiniBand?<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Modern InfiniBand (e.g., NDR) reaches 400 Gbps per port, with sub-microsecond latency, making it one of the fastest interconnects available.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-why-is-infiniband-so-fast\"><span class=\"ez-toc-section\" id=\"Why_Is_InfiniBand_So_Fast\"><\/span><strong>Why Is InfiniBand So Fast?<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>InfiniBand&#8217;s speed stems from its switched fabric architecture, RDMA support, and optimized protocols, minimizing overhead and maximizing throughput.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-who-uses-infiniband\"><span class=\"ez-toc-section\" id=\"Who_Uses_InfiniBand\"><\/span><strong>Who Uses InfiniBand?<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>InfiniBand is used by HPC centers, AI research labs (e.g., Meta AI, Google DeepMind), cloud providers, and scientific institutions for high-performance tasks.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-how-do-i-optimize-my-network-for-ai-workloads\"><span class=\"ez-toc-section\" id=\"How_Do_I_Optimize_My_Network_for_AI_Workloads\"><\/span><strong>How Do I Optimize My Network for AI Workloads?<\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Use InfiniBand for GPU clusters, optimize topology (e.g., fat-tree), ensure high-quality cabling, and leverage RDMA for low-latency data transfers.<\/p>\n\n\n\n<p><\/p>\n<style>\r\n\r\n        .lwrp.link-whisper-related-posts{\r\n            \r\n            margin-top: 40px;\nmargin-bottom: 30px;\r\n        }\r\n        .lwrp .lwrp-title{\r\n            \r\n            \r\n        }\r\n        .lwrp .lwrp-description{\r\n            \r\n            \r\n\r\n        }\r\n        .lwrp .lwrp-list-container{\r\n        }\r\n        .lwrp .lwrp-list-multi-container{\r\n            display: flex;\r\n        }\r\n        .lwrp .lwrp-list-double{\r\n            width: 48%;\r\n        }\r\n        .lwrp .lwrp-list-triple{\r\n            width: 32%;\r\n        }\r\n        .lwrp .lwrp-list-row-container{\r\n            display: flex;\r\n            justify-content: space-between;\r\n        }\r\n        .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n            width: calc(100% - 20px);\r\n        }\r\n        .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n            \r\n            list-style: decimal;\r\n        }\r\n        .lwrp .lwrp-list-item img{\r\n            max-width: 100%;\r\n            height: auto;\r\n        }\r\n        .lwrp .lwrp-list-item.lwrp-empty-list-item{\r\n            background: initial !important;\r\n        }\r\n        .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n        .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n            \r\n                \r\n        }\r\n        @media screen and (max-width: 480px) {\r\n            .lwrp.link-whisper-related-posts{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-title{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-description{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-multi-container{\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-multi-container ul.lwrp-list{\r\n                margin-top: 0px;\r\n                margin-bottom: 0px;\r\n                padding-top: 0px;\r\n                padding-bottom: 0px;\r\n            }\r\n            .lwrp .lwrp-list-double,\r\n            .lwrp .lwrp-list-triple{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-row-container{\r\n                justify-content: initial;\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n            .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n                \r\n                    \r\n            }\r\n        }<\/style>\r\n<div id=\"link-whisper-related-posts-widget\" class=\"link-whisper-related-posts lwrp\">\r\n            <h3 class=\"lwrp-title\">Related Posts<\/h3>    \r\n        <div class=\"lwrp-list-container\">\r\n                                            <ul class=\"lwrp-list lwrp-list-single\">\r\n                    <li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/fiber-optic-hdmi-cables.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Why Fiber Optic HDMI Cables are Preferred over Common HDMI Cables<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/sfp28-nic.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Unveiling the Power of SFP28 NIC: Your Ultimate Guide to Network Interface Card<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/video\/sfp28-er30-vs-sfp28-er-transceiver.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">SFP28 ER30 vs. SFP28 ER transceiver: what&#8217;s the difference?<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/mellanox-infiniband-adapter.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Mellanox\u00ae InfiniBand Adapter: Ultimate Guide to High-Performance Network Solutions<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/data-switch.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Ultimate Guide to Data Switch: Ensuring Seamless Data Transfer for Network Efficiency<\/span><\/a><\/li>                <\/ul>\r\n                        <\/div>\r\n<\/div>","protected":false},"excerpt":{"rendered":"<p>History and Standards InfiniBand, developed in the late 1990s, was designed for high-speed, low-latency interconnects in HPC environments. It operates under standards set by the InfiniBand Trade Association (IBTA), with modern versions like HDR and NDR supporting speeds up to 400 Gbps. Ethernet, dating back to the 1970s, is governed by the IEEE 802.3 standards [&hellip;]<\/p>\n","protected":false},"author":5,"featured_media":1442,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"site-sidebar-layout":"default","site-content-layout":"default","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"default","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":"","_wpscppro_dont_share_socialmedia":false,"_wpscppro_custom_social_share_image":0,"_facebook_share_type":"default","_twitter_share_type":"default","_linkedin_share_type":"default","_pinterest_share_type":"default","_linkedin_share_type_page":"default","_instagram_share_type":"default","_medium_share_type":"default","_threads_share_type":"default","_google_business_share_type":"default","_selected_social_profile":[{"id":"skM9ewvR8O","platform":"linkedin","platformKey":0,"name":"Jason Xue","type":"person","thumbnail_url":"https:\/\/media.licdn.com\/dms\/image\/C5603AQErPqKD0j6qBg\/profile-displayphoto-shrink_100_100\/0\/1599138392315?e=1723075200&v=beta&t=joEkh1OeKQ0F-QpAPv4xxQyBdGlHyccIQZauRSs6RvU","share_type":"default"}],"_wpsp_enable_custom_social_template":false,"_wpsp_social_scheduling":{"enabled":false,"datetime":null,"platforms":[],"status":"template_only","dateOption":"today","timeOption":"now","customDays":"","customHours":"","customDate":"","customTime":"","schedulingType":"absolute"},"_wpsp_active_default_template":true},"categories":[2,29],"tags":[],"class_list":["post-1437","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-networking"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v20.13 (Yoast SEO v25.8) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>What is InfiniBand and its difference with Ethernet | FiberMall<\/title>\n<meta name=\"description\" content=\"This article will introduce the InfiniBand vs Ethernet and what is InfiniBand network and 200G InfiniBand HDR.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What is InfiniBand Network and the Difference with Ethernet?\" \/>\n<meta property=\"og:description\" content=\"History and Standards InfiniBand, developed in the late 1990s, was designed for high-speed, low-latency interconnects in HPC environments. It operates\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\" \/>\n<meta property=\"og:site_name\" content=\"fibermall.com\" \/>\n<meta property=\"article:published_time\" content=\"2021-12-01T08:51:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-13T09:17:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"855\" \/>\n\t<meta property=\"og:image:height\" content=\"504\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Brian\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Brian\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"22 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\"},\"author\":{\"name\":\"Brian\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/9421292f8817d668e9e64f94dfe9b901\"},\"headline\":\"What is InfiniBand Network and the Difference with Ethernet?\",\"datePublished\":\"2021-12-01T08:51:08+00:00\",\"dateModified\":\"2026-03-13T09:17:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\"},\"wordCount\":4370,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg\",\"articleSection\":[\"Blog\",\"Networking\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\",\"url\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\",\"name\":\"What is InfiniBand and its difference with Ethernet | FiberMall\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg\",\"datePublished\":\"2021-12-01T08:51:08+00:00\",\"dateModified\":\"2026-03-13T09:17:14+00:00\",\"description\":\"This article will introduce the InfiniBand vs Ethernet and what is InfiniBand network and 200G InfiniBand HDR.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg\",\"width\":855,\"height\":504,\"caption\":\"fiber mall infiniband\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fibermall.com\/blog.htm\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What is InfiniBand Network and the Difference with Ethernet?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"name\":\"fibermall.com\",\"description\":\"Optical Communication Expert\",\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\",\"name\":\"fibermall.com\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"width\":200,\"height\":67,\"caption\":\"fibermall.com\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/9421292f8817d668e9e64f94dfe9b901\",\"name\":\"Brian\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c196835c5ddcc99559fcfaa0046ffbd895891b78d40ab46bf34f7a5ecd73dfd9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c196835c5ddcc99559fcfaa0046ffbd895891b78d40ab46bf34f7a5ecd73dfd9?s=96&d=mm&r=g\",\"caption\":\"Brian\"},\"description\":\"Optical Network Engineer\",\"sameAs\":[\"https:\/\/www.fibermall.com\"],\"url\":\"https:\/\/www.fibermall.com\/blog.htm?author=5\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"What is InfiniBand and its difference with Ethernet | FiberMall","description":"This article will introduce the InfiniBand vs Ethernet and what is InfiniBand network and 200G InfiniBand HDR.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm","og_locale":"en_US","og_type":"article","og_title":"What is InfiniBand Network and the Difference with Ethernet?","og_description":"History and Standards InfiniBand, developed in the late 1990s, was designed for high-speed, low-latency interconnects in HPC environments. It operates","og_url":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm","og_site_name":"fibermall.com","article_published_time":"2021-12-01T08:51:08+00:00","article_modified_time":"2026-03-13T09:17:14+00:00","og_image":[{"width":855,"height":504,"url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg","type":"image\/jpeg"}],"author":"Brian","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Brian","Est. reading time":"22 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#article","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm"},"author":{"name":"Brian","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/9421292f8817d668e9e64f94dfe9b901"},"headline":"What is InfiniBand Network and the Difference with Ethernet?","datePublished":"2021-12-01T08:51:08+00:00","dateModified":"2026-03-13T09:17:14+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm"},"wordCount":4370,"commentCount":0,"publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg","articleSection":["Blog","Networking"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm","url":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm","name":"What is InfiniBand and its difference with Ethernet | FiberMall","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg","datePublished":"2021-12-01T08:51:08+00:00","dateModified":"2026-03-13T09:17:14+00:00","description":"This article will introduce the InfiniBand vs Ethernet and what is InfiniBand network and 200G InfiniBand HDR.","breadcrumb":{"@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#primaryimage","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2021\/12\/front-image.jpg","width":855,"height":504,"caption":"fiber mall infiniband"},{"@type":"BreadcrumbList","@id":"https:\/\/www.fibermall.com\/blog\/what-is-infiniband-network-and-difference-with-ethernet.htm#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fibermall.com\/blog.htm"},{"@type":"ListItem","position":2,"name":"What is InfiniBand Network and the Difference with Ethernet?"}]},{"@type":"WebSite","@id":"https:\/\/www.fibermall.com\/blog.htm\/#website","url":"https:\/\/www.fibermall.com\/blog.htm\/","name":"fibermall.com","description":"Optical Communication Expert","publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization","name":"fibermall.com","url":"https:\/\/www.fibermall.com\/blog.htm\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","width":200,"height":67,"caption":"fibermall.com"},"image":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/9421292f8817d668e9e64f94dfe9b901","name":"Brian","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c196835c5ddcc99559fcfaa0046ffbd895891b78d40ab46bf34f7a5ecd73dfd9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c196835c5ddcc99559fcfaa0046ffbd895891b78d40ab46bf34f7a5ecd73dfd9?s=96&d=mm&r=g","caption":"Brian"},"description":"Optical Network Engineer","sameAs":["https:\/\/www.fibermall.com"],"url":"https:\/\/www.fibermall.com\/blog.htm?author=5"}]}},"_links":{"self":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/1437","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1437"}],"version-history":[{"count":14,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/1437\/revisions"}],"predecessor-version":[{"id":18734,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/1437\/revisions\/18734"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/media\/1442"}],"wp:attachment":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1437"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1437"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1437"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}