{"id":12827,"date":"2024-09-05T07:08:08","date_gmt":"2024-09-05T07:08:08","guid":{"rendered":"https:\/\/www.fibermall.com\/blog\/?p=12827"},"modified":"2024-09-05T09:36:13","modified_gmt":"2024-09-05T09:36:13","slug":"ib-network-and-mellanox-products","status":"publish","type":"post","link":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm","title":{"rendered":"IB Network and Mellanox Products"},"content":{"rendered":"\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#What_is_IB-InfiniBand\" >What is IB-InfiniBand<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#Core_advantages_of_Mellanox_and_IB_Network\" >Core advantages of Mellanox and IB Network<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#IB_Network_and_RDMA\" >IB Network and RDMA<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#Comparison_of_two_mainstream_RDMA_solutions_IB_and_ROCEV2\" >Comparison of two mainstream RDMA solutions (IB and ROCEV2)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#Development_and_Trend_of_InfiniBand_Link_Rate\" >Development and Trend of InfiniBand Link Rate<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#NVIDIA_InfiniBand_mainstream_products_%E2%80%93_the_latest_NDR_network_card_situation\" >NVIDIA InfiniBand mainstream products &#8211; the latest NDR network card situation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#Mellanoxs_latest_NDR_switch\" >Mellanox&#8217;s latest NDR switch<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#Mellanoxs_latest_interconnect_cables_and_modules\" >Mellanox&#8217;s latest interconnect cables and modules<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\/#Mellanox_Typical_Interconnect_Links\" >Mellanox Typical Interconnect Links<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\" id=\"h-what-is-ib-infiniband\"><span class=\"ez-toc-section\" id=\"What_is_IB-InfiniBand\"><\/span>What is IB-InfiniBand<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>IB, the abbreviation of InfiniBand (translated as &#8221;&nbsp;<strong>Infinitive bandwidth<\/strong>&nbsp;\u201d ), is a computer network communication standard for high-performance computing. IB technology, characterized by extremely high throughput and extremely low latency, is used for data interconnection between computers. InfiniBand is also used as a direct or switched interconnect between servers and storage systems, and as an interconnect between storage systems. With the rise of AI, IB has also become popular and is currently the preferred networking method for high-end GPU server clusters.<\/p>\n\n\n\n<p>Here is the development history of InfiniBand:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>1999: The InfiniBand Trade Association (IBTA) released the InfiniBand architecture, which was originally intended to replace the PCI bus.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In 2000, version 1.0 of the InfiniBand architecture specification was officially released. Then in 2001, the first batch of InfiniBand products came out, and many manufacturers began to launch products that support InfiniBand, including servers, storage systems, and network equipment.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In 2003, thanks to its high throughput and low latency, InfiniBand turned to a new application area &#8211; HPC computer cluster interconnection, and was widely used in the TOP500 supercomputers at the time.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In 2004, another important InfiniBand non-profit organization was born &#8211; Open Fabrics Alliance (OFA ).<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In 2005, InfiniBand found a new scenario &#8211; the connection of storage devices, and has been continuously updated and iterated since then.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In 2015, InfiniBand technology accounted for more than 50% of TOP500 supercomputers for the first time, reaching 51.4%. This marks the first time that InfiniBand technology has overtaken Ethernet technology to become the most popular internal connection technology in supercomputers.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Since 2023, AI large model training has been highly dependent on high-performance computing clusters, and InfiniBand networks are the best partner for high-performance computing clusters.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-core-advantages-of-mellanox-and-ib-network\"><span class=\"ez-toc-section\" id=\"Core_advantages_of_Mellanox_and_IB_Network\"><\/span>Core advantages of Mellanox and IB Network<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><strong>Relationship between Mellanox and InfiniBand<\/strong><\/p>\n\n\n\n<p>Currently, when people mention IB, the first thing that comes to mind is Mellanox. In 2019, NVIDIA spent US$6.9 billion to acquire Mellanox as its sub-brand. Jensen Huang said publicly: This is a combination of two of the world&#8217;s leading high-performance computing companies. NVIDIA focuses on accelerated computing, and Mellanox focuses on interconnection and storage.<\/p>\n\n\n\n<p>According to forecasts by industry organizations, the market size of <a href=\"https:\/\/www.fibermall.com\/collection\/infiniband.htm\" target=\"_blank\" rel=\"noreferrer noopener\">InfiniBand<\/a> will reach US$98.37 billion in 2029, a 14.7-fold increase from US$6.66 billion in 2021. Driven by high-performance computing and AI, InfiniBand has a bright future.<\/p>\n\n\n\n<p><strong>&nbsp;InfiniBand network architecture and features<\/strong><\/p>\n\n\n\n<p>An InfiniBand system consists of channel adapters, switches, routers, cables, and connectors, as shown in the following figure.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img fetchpriority=\"high\" decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/InfiniBand-Systems.png\" alt=\"InfiniBand System\" class=\"wp-image-12831\" style=\"width:456px;height:295px\" width=\"456\" height=\"295\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/InfiniBand-Systems.png 970w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/InfiniBand-Systems-300x194.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/InfiniBand-Systems-768x496.png 768w\" sizes=\"(max-width: 456px) 100vw, 456px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><strong>InfiniBand System<\/strong><\/p>\n\n\n\n<p>The core features are summarized as follows:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Low latency: extremely low latency and native support for RDMA<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>High bandwidth: 400Gb\/s data transmission capability per port<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ease of use: Suitable for building large-scale data center clusters<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-ib-network-and-rdma\"><span class=\"ez-toc-section\" id=\"IB_Network_and_RDMA\"><\/span>IB Network and RDMA<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>When talking about IB network, we have to mention RDMA. RDMA (Remote Direct Memory Access) was created to solve the delay of server-side data processing in network transmission. It can directly access the memory of another host or server from the memory of one host or server without using the CPU. It frees up the CPU to perform its work. Infiniband is a network technology designed specifically for RDMA, and IB network natively supports RDMA.<\/p>\n\n\n\n<p>The reason why RDMA technology is so powerful is its kernel bypass mechanism, which allows direct data reading and writing between applications and network cards, reducing the data transmission latency within the server to nearly 1us. Roce transplants RDMA to Ethernet.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-comparison-of-two-mainstream-rdma-solutions-ib-and-rocev2\"><span class=\"ez-toc-section\" id=\"Comparison_of_two_mainstream_RDMA_solutions_IB_and_ROCEV2\"><\/span>Comparison of two mainstream RDMA solutions (IB and ROCEV2)<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Compared with traditional data centers, the new intelligent computing center communication network requirements are higher, such as low latency, large bandwidth, stability, and large scale.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>RDMA-based InfiniBand and ROCEV2 can both meet the needs of intelligent computing center communication networks.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>InfiniBand currently has more advantages in performance than ROCEV2, while ROCEV2 currently has advantages over InfiniBand in terms of economy and versatility.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-development-and-trend-of-infiniband-link-rate\"><span class=\"ez-toc-section\" id=\"Development_and_Trend_of_InfiniBand_Link_Rate\"><\/span>Development and Trend of InfiniBand Link Rate<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Taking the early SDR (single data rate) specification as an example, the original signal bandwidth of a 1X link is 2.5Gbps, a 4X link is 10Gbps, and a 12X link is 30Gbps. The actual data bandwidth of a 1X link is 2.0 Gbps (due to 8b\/10b encoding). Since the link is bidirectional, the total bandwidth relative to the bus is 4Gbps. Over time, the network bandwidth of InfiniBand continues to upgrade.<\/p>\n\n\n\n<p>What do the DRs mean in HDR and NDR? Each DR stands for the abbreviation of each generation of IB technology. DR is a general term for data rate, with 4 channels being the mainstream.<\/p>\n\n\n\n<p>The following figure shows the network bandwidth of InfiniBand from SDR, DDR, QDR, FDR, EDR to HDR and NDR. The speed is based on 4x link speed. Currently, EDR, HDR and NDR are mainstream, corresponding to PICE 3.0, 4.0 and 5.0 server platforms.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-nvidia-infiniband-mainstream-products-the-latest-ndr-network-card-situation\"><span class=\"ez-toc-section\" id=\"NVIDIA_InfiniBand_mainstream_products_%E2%80%93_the_latest_NDR_network_card_situation\"><\/span>NVIDIA InfiniBand mainstream products &#8211; the latest NDR network card situation<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>ConnectX-7 IB Card (HCA) has a variety of form factors, including single and dual ports, supporting OSFP and QSFP112 interfaces, and supporting 200Gbps and 400Gbps rates. The CX-7 network card supports x16 PCle5.0 or PCle 4.0, &nbsp;which complies with CEM specifications. Up to 16 lanes can be connected with support for an optional auxiliary card that enables 32 lanes of PCIe 4.0 using NVIDIA Socket Direct\u00ae technology.<\/p>\n\n\n\n<p>Other form factors include Open Compute Project (OCP) 3.0 with an OSFP connector, OCP 3.0 with a QSFP112 connector, and CEM PCle x16 with a QSFP112 connector.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-mellanox-s-latest-ndr-switch\"><span class=\"ez-toc-section\" id=\"Mellanoxs_latest_NDR_switch\"><\/span>Mellanox&#8217;s latest NDR switch<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Mellanox&#8217;s IB switches are divided into two types: fixed configuration switches and modular switches. It is understood that the latest NDR series switches no longer sell modular configuration switches (although the official website shows that they are available, they are no longer on sale).<\/p>\n\n\n\n<p>NDR&#8217;s fixed configuration switch MQM9700 series is equipped with 32 physical OSFP connectors and supports 64 400Gb\/s ports (which can be split into up to 128 200Gb\/s ports). The switch series provides a total of 51.2Tb\/s of bidirectional throughput ( backplane bandwidth) and an astonishing 66.5 billion packets per second (packet forwarding rate).<\/p>\n\n\n\n<p>The number of interfaces and speeds of the sub-models are the same, and the differences lie in whether the management function is supported, in the power supply method and the heat dissipation method. Usually one switch with management functions is sufficient.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-mellanox-s-latest-interconnect-cables-and-modules\"><span class=\"ez-toc-section\" id=\"Mellanoxs_latest_interconnect_cables_and_modules\"><\/span>Mellanox&#8217;s latest interconnect cables and modules<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Mellanox&#8217;s LinkX cables and transceivers are typically used to link ToR switches downward to NVIDIA GPU and CPU server network cards and storage devices, and\/or upward in switch-to-switch interconnect applications throughout the network infrastructure.<\/p>\n\n\n\n<p>Active Optical Cable (AOC), Direct Attach Copper Cable (DAC), and the new active DAC called ACC, which include a signal enhancement integrated circuit (IC) at the end of the cable.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-Latest-Interconnect-Cables-and-Modules-1024x350.png\" alt=\"\" class=\"wp-image-12837\" style=\"width:786px;height:269px\" width=\"786\" height=\"269\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-Latest-Interconnect-Cables-and-Modules-1024x350.png 1024w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-Latest-Interconnect-Cables-and-Modules-300x103.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-Latest-Interconnect-Cables-and-Modules-768x263.png 768w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-Latest-Interconnect-Cables-and-Modules.png 1269w\" sizes=\"(max-width: 786px) 100vw, 786px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><strong>Mellanox Latest Interconnect Cables and Modules<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-mellanox-typical-interconnect-links\"><span class=\"ez-toc-section\" id=\"Mellanox_Typical_Interconnect_Links\"><\/span>Mellanox Typical Interconnect Links<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>Switches to switches and switches to network cards can be interconnected through different cables, and switch to network card can achieve 1-to-2 or 4-to-1 interconnection.<\/p>\n\n\n\n<p><strong>Mellanox NIC topology in H100<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The HGX module is logically connected to the head through 4 or 8 PCIE SW chips inside the H100 machine.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Each PCIE sw corresponds to two GPU cards and two network cards, and the eight 400G IB cards are designed to correspond one-to-one with the eight H100 cards.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If it is fully equipped with eight 400G IB cards, it will require other PCIE SW connections from the CPU to add other network cards.<\/li>\n<\/ul>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100.png\" alt=\"\" class=\"wp-image-12838\" style=\"width:572px;height:333px\" width=\"572\" height=\"333\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100.png 696w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-300x175.png 300w\" sizes=\"(max-width: 572px) 100vw, 572px\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center\"><strong>Mellanox NIC topology in H100<\/strong><\/p>\n<style>\r\n\r\n        .lwrp.link-whisper-related-posts{\r\n            \r\n            margin-top: 40px;\nmargin-bottom: 30px;\r\n        }\r\n        .lwrp .lwrp-title{\r\n            \r\n            \r\n        }\r\n        .lwrp .lwrp-description{\r\n            \r\n            \r\n\r\n        }\r\n        .lwrp .lwrp-list-container{\r\n        }\r\n        .lwrp .lwrp-list-multi-container{\r\n            display: flex;\r\n        }\r\n        .lwrp .lwrp-list-double{\r\n            width: 48%;\r\n        }\r\n        .lwrp .lwrp-list-triple{\r\n            width: 32%;\r\n        }\r\n        .lwrp .lwrp-list-row-container{\r\n            display: flex;\r\n            justify-content: space-between;\r\n        }\r\n        .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n            width: calc(100% - 20px);\r\n        }\r\n        .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n            \r\n            list-style: decimal;\r\n        }\r\n        .lwrp .lwrp-list-item img{\r\n            max-width: 100%;\r\n            height: auto;\r\n        }\r\n        .lwrp .lwrp-list-item.lwrp-empty-list-item{\r\n            background: initial !important;\r\n        }\r\n        .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n        .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n            \r\n                \r\n        }\r\n        @media screen and (max-width: 480px) {\r\n            .lwrp.link-whisper-related-posts{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-title{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-description{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-multi-container{\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-multi-container ul.lwrp-list{\r\n                margin-top: 0px;\r\n                margin-bottom: 0px;\r\n                padding-top: 0px;\r\n                padding-bottom: 0px;\r\n            }\r\n            .lwrp .lwrp-list-double,\r\n            .lwrp .lwrp-list-triple{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-row-container{\r\n                justify-content: initial;\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n            .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n                \r\n                    \r\n            }\r\n        }<\/style>\r\n<div id=\"link-whisper-related-posts-widget\" class=\"link-whisper-related-posts lwrp\">\r\n            <h3 class=\"lwrp-title\">Related Posts<\/h3>    \r\n        <div class=\"lwrp-list-container\">\r\n                                            <ul class=\"lwrp-list lwrp-list-single\">\r\n                    <li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/video\/the-use-of-glc-sx-mm.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">5 things you need to know about GLC-SX-MM<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/network-adapter-nics-function-construction-classification.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">What is Network Adapter: Function, Construction and Classification of NICs<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/hdr-infiniband.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Understanding HDR Infiniband: A Comprehensive Guide to Modern Networking<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/10-port-switch.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Unveiling the Best 10-Port Gigabit Ethernet Switch for Your Network Needs<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/questions\/what-form-factors-used-for-400g-optics.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">What Form Factors Are Used for 400G Optics and Cables?<\/span><\/a><\/li>                <\/ul>\r\n                        <\/div>\r\n<\/div>","protected":false},"excerpt":{"rendered":"<p>What is IB-InfiniBand IB, the abbreviation of InfiniBand (translated as &#8221;&nbsp;Infinitive bandwidth&nbsp;\u201d ), is a computer network communication standard for high-performance computing. IB technology, characterized by extremely high throughput and extremely low latency, is used for data interconnection between computers. InfiniBand is also used as a direct or switched interconnect between servers and storage systems, [&hellip;]<\/p>\n","protected":false},"author":227,"featured_media":12840,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":"","_wpscppro_dont_share_socialmedia":false,"_wpscppro_custom_social_share_image":0,"_facebook_share_type":"default","_twitter_share_type":"default","_linkedin_share_type":"default","_pinterest_share_type":"default","_linkedin_share_type_page":"default","_instagram_share_type":"default","_medium_share_type":"","_threads_share_type":"","_google_business_share_type":"","_selected_social_profile":[{"id":"skM9ewvR8O","platform":"linkedin","platformKey":0,"name":"Jason Xue","type":"person","thumbnail_url":"https:\/\/media.licdn.com\/dms\/image\/C5603AQErPqKD0j6qBg\/profile-displayphoto-shrink_100_100\/0\/1599138392315?e=1723075200&v=beta&t=joEkh1OeKQ0F-QpAPv4xxQyBdGlHyccIQZauRSs6RvU","share_type":"default"}],"_wpsp_enable_custom_social_template":false,"_wpsp_social_scheduling":{"enabled":false,"datetime":null,"platforms":[],"status":"template_only","dateOption":"today","timeOption":"now","customDays":"","customHours":"","customDate":"","customTime":"","schedulingType":"absolute"},"_wpsp_active_default_template":true},"categories":[2,29],"tags":[],"class_list":["post-12827","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-networking"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v20.13 (Yoast SEO v25.8) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>IB Network and Mellanox Products | FiberMall<\/title>\n<meta name=\"description\" content=\"Mellanox&#039;s LinkX cables and transceivers are typically used to link ToR switches downward to NVIDIA GPU and CPU server network cards.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"IB Network and Mellanox Products\" \/>\n<meta property=\"og:description\" content=\"What is IB-InfiniBand IB, the abbreviation of InfiniBand (translated as &quot;&nbsp;Infinitive bandwidth&nbsp;\u201d ), is a computer network communication standard\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\" \/>\n<meta property=\"og:site_name\" content=\"fibermall.com\" \/>\n<meta property=\"article:published_time\" content=\"2024-09-05T07:08:08+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-05T09:36:13+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png\" \/>\n\t<meta property=\"og:image:width\" content=\"600\" \/>\n\t<meta property=\"og:image:height\" content=\"390\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Catherine\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Catherine\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\"},\"author\":{\"name\":\"Catherine\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/bd2cf8d53f883c88f6ebaa5c3ed1258d\"},\"headline\":\"IB Network and Mellanox Products\",\"datePublished\":\"2024-09-05T07:08:08+00:00\",\"dateModified\":\"2024-09-05T09:36:13+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\"},\"wordCount\":1211,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png\",\"articleSection\":[\"Blog\",\"Networking\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\",\"url\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\",\"name\":\"IB Network and Mellanox Products | FiberMall\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png\",\"datePublished\":\"2024-09-05T07:08:08+00:00\",\"dateModified\":\"2024-09-05T09:36:13+00:00\",\"description\":\"Mellanox's LinkX cables and transceivers are typically used to link ToR switches downward to NVIDIA GPU and CPU server network cards.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png\",\"width\":600,\"height\":390,\"caption\":\"Mellanox NIC topology in H100\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fibermall.com\/blog.htm\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"IB Network and Mellanox Products\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"name\":\"fibermall.com\",\"description\":\"Optical Communication Expert\",\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\",\"name\":\"fibermall.com\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"width\":200,\"height\":67,\"caption\":\"fibermall.com\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/bd2cf8d53f883c88f6ebaa5c3ed1258d\",\"name\":\"Catherine\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/3a11235a453567962b9c9bb843b31bb4396259dd8abad3f3c3758babd287d778?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/3a11235a453567962b9c9bb843b31bb4396259dd8abad3f3c3758babd287d778?s=96&d=mm&r=g\",\"caption\":\"Catherine\"},\"description\":\"Optical Communications Engineer\",\"sameAs\":[\"https:\/\/www.fibermall.com\/\"],\"url\":\"https:\/\/www.fibermall.com\/blog.htm?author=227\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"IB Network and Mellanox Products | FiberMall","description":"Mellanox's LinkX cables and transceivers are typically used to link ToR switches downward to NVIDIA GPU and CPU server network cards.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm","og_locale":"en_US","og_type":"article","og_title":"IB Network and Mellanox Products","og_description":"What is IB-InfiniBand IB, the abbreviation of InfiniBand (translated as \"&nbsp;Infinitive bandwidth&nbsp;\u201d ), is a computer network communication standard","og_url":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm","og_site_name":"fibermall.com","article_published_time":"2024-09-05T07:08:08+00:00","article_modified_time":"2024-09-05T09:36:13+00:00","og_image":[{"width":600,"height":390,"url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png","type":"image\/png"}],"author":"Catherine","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Catherine","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#article","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm"},"author":{"name":"Catherine","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/bd2cf8d53f883c88f6ebaa5c3ed1258d"},"headline":"IB Network and Mellanox Products","datePublished":"2024-09-05T07:08:08+00:00","dateModified":"2024-09-05T09:36:13+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm"},"wordCount":1211,"commentCount":0,"publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png","articleSection":["Blog","Networking"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm","url":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm","name":"IB Network and Mellanox Products | FiberMall","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png","datePublished":"2024-09-05T07:08:08+00:00","dateModified":"2024-09-05T09:36:13+00:00","description":"Mellanox's LinkX cables and transceivers are typically used to link ToR switches downward to NVIDIA GPU and CPU server network cards.","breadcrumb":{"@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fibermall.com\/blog\/mellanox-products.htm"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#primaryimage","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/09\/Mellanox-NIC-topology-in-H100-fi.png","width":600,"height":390,"caption":"Mellanox NIC topology in H100"},{"@type":"BreadcrumbList","@id":"https:\/\/www.fibermall.com\/blog\/mellanox-products.htm#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fibermall.com\/blog.htm"},{"@type":"ListItem","position":2,"name":"IB Network and Mellanox Products"}]},{"@type":"WebSite","@id":"https:\/\/www.fibermall.com\/blog.htm\/#website","url":"https:\/\/www.fibermall.com\/blog.htm\/","name":"fibermall.com","description":"Optical Communication Expert","publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization","name":"fibermall.com","url":"https:\/\/www.fibermall.com\/blog.htm\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","width":200,"height":67,"caption":"fibermall.com"},"image":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/bd2cf8d53f883c88f6ebaa5c3ed1258d","name":"Catherine","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/3a11235a453567962b9c9bb843b31bb4396259dd8abad3f3c3758babd287d778?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/3a11235a453567962b9c9bb843b31bb4396259dd8abad3f3c3758babd287d778?s=96&d=mm&r=g","caption":"Catherine"},"description":"Optical Communications Engineer","sameAs":["https:\/\/www.fibermall.com\/"],"url":"https:\/\/www.fibermall.com\/blog.htm?author=227"}]}},"_links":{"self":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/12827","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/users\/227"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=12827"}],"version-history":[{"count":8,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/12827\/revisions"}],"predecessor-version":[{"id":12881,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/12827\/revisions\/12881"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/media\/12840"}],"wp:attachment":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=12827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=12827"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=12827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}