{"id":11818,"date":"2024-08-09T08:37:14","date_gmt":"2024-08-09T08:37:14","guid":{"rendered":"https:\/\/www.fibermall.com\/blog\/?p=11818"},"modified":"2024-08-09T08:37:28","modified_gmt":"2024-08-09T08:37:28","slug":"unleashing-performance-the-power-of-infiniband","status":"publish","type":"post","link":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm","title":{"rendered":"Unleashing Performance: The Power of InfiniBand"},"content":{"rendered":"\n<p>The effective computational power of a cluster can be broken down into GPU utilization and the cluster\u2019s linear speedup. GPU utilization is influenced by factors such as chip architecture, process technology, memory, I\/O bottlenecks, inter-card interconnect bandwidth, topology, and power consumption. On the other hand, \u2018cluster linear speedup\u2019 depends on node communication capabilities, parallel training frameworks, and resource scheduling.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/CPU-1024x483.png\" alt=\"CPU\" class=\"wp-image-11827\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/CPU-1024x483.png 1024w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/CPU-300x141.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/CPU-768x362.png 768w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/CPU.png 1078w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Designing an efficient cluster networking solution is crucial to achieve low latency, high bandwidth, and unblocked inter-node communication. This reduces communication overhead between multiple machines and GPUs, ultimately improving the effective GPU compute time (GPU compute time \/ overall training time). Based on the \u2018White Paper on Network Evolution for AI Large Models\u2019 by China Mobile Research Institute, AI large models pose new requirements for network infrastructure:<\/p>\n\n\n\n<ol class=\"wp-block-list\" type=\"1\">\n<li>Ultra-Large-Scale Networking: AI super-large models with parameter counts reaching billions to trillions demand extremely high computational power. Consequently, this necessitates a large amount of hardware and scalable network capacity. According to the \u2018White Paper on Novel Intelligent Computing Technologies for Ultra-Thousand-Card Clusters,\u2019 achieving optimal computational efficiency, data processing capabilities, hardware-level interconnection for thousands of cards, and network availability becomes a critical topic for AI computing centers.<\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top-1.png\" alt=\"top\" class=\"wp-image-11828\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top-1.png 886w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top-1-300x174.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top-1-768x445.png 768w\" sizes=\"(max-width: 886px) 100vw, 886px\" \/><\/figure>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ultra-High Bandwidth Requirements: In multi-GPU clusters, both intra-server and inter-server communication are inevitable. Intra-server communication involves All Reduce collective communication data generated by model parallelism, reaching hundreds of gigabytes. Thus, the intra-GPU communication bandwidth and method significantly impact end-to-end flow completion time. Inter-server communication, in modes like pipeline parallelism, data parallelism, and tensor parallelism, also reaches similar data volumes. Complex collective communication patterns simultaneously involve many-to-one and one-to-many communication. Therefore, high-speed interconnects between GPUs are essential for single-port bandwidth, available inter-node links, and overall network bandwidth.<\/li>\n\n\n\n<li>Ultra-Low Latency: Data communication latency consists of static and dynamic components. Static latency depends on forwarding chip capabilities and transmission distance. When network topology and communication data volume are fixed, this part of the latency remains relatively constant. Dynamic latency includes switch internal queuing delay and packet loss retransmission delay, typically caused by network congestion, packet loss, and jitter.<\/li>\n\n\n\n<li>Ultra-High Stability and Automated Deployment: With a significant increase in the number of cards, network stability becomes the \u2018weakest link\u2019 in cluster networks. Network failures and performance fluctuations impact both inter-node connectivity and resource utilization.<\/li>\n<\/ul>\n\n\n\n<p>RDMA (Remote Direct Memory Access) can reduce end-to-end communication latency between multiple machines and GPUs. In traditional networks, data transfer involves several steps: first, copying data from the source system\u2019s kernel to the network stack, then transmitting it over the network. Finally, at the receiving end, multiple steps copy the data back to the target system\u2019s kernel. RDMA bypasses the operating system kernel, allowing one host to directly access another host\u2019s memory. Currently, the main RDMA technologies are InfiniBand and RoCEv2 (RDMA over Converged Ethernet).<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/traditional-and-rdma-1.png\" alt=\"traditional and rdma\" class=\"wp-image-11829\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/traditional-and-rdma-1.png 915w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/traditional-and-rdma-1-300x179.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/traditional-and-rdma-1-768x459.png 768w\" sizes=\"(max-width: 915px) 100vw, 915px\" \/><\/figure>\n\n\n\n<p>InfiniBand (IB) and RDMA over Converged Ethernet (RoCE) are two prominent network technologies. IB excels in low latency and high bandwidth, while Ethernet offers openness and cost-effectiveness. RoCE, being widely adopted and mature, serves as a cornerstone for interconnecting different systems with good compatibility. It also benefits from multiple vendors, resulting in cost advantages.<\/p>\n\n\n\n<p>In contrast, IB is specialized for high-bandwidth, low-latency, and reliable network interconnects, commonly used in HPC clusters. However, due to limited vendor support, its deployment cost is higher than RoCE.<\/p>\n\n\n\n<p>RoCE is a solid choice, while InfiniBand stands out as an exceptional solution. Notably, in supercomputing clusters, IB remains a popular and efficient interconnect. Nevertheless, considering cost and openness, many cloud computing companies opt for open-source Ethernet switches over proprietary IB solutions. According to Brian Barrett, a senior engineer at AWS, dedicated IB networks can be like isolated islands in the vast ocean of flexible resource allocation and sharing within cloud data centers.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top500-1.png\" alt=\"top500\" class=\"wp-image-11830\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top500-1.png 843w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top500-1-300x193.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/top500-1-768x495.png 768w\" sizes=\"(max-width: 843px) 100vw, 843px\" \/><\/figure>\n\n\n\n<p>Additionally, the Ultra Ethernet Consortium (UEC) was established on July 19, 2023, under the Linux Foundation\u2019s leadership. Comprising cloud providers (such as MATA and Microsoft), network equipment manufacturers (including Broadcom, Cisco, and HP), and semiconductor companies (AMD and Intel), UEC aims to provide an open, interoperable, high-performance communication stack based on Ethernet. This initiative supports the growing network demands of AI and HPC.<\/p>\n\n\n\n<p>InfiniBand, with its early adoption of RDMA, offers native advantages such as low latency, high bandwidth, and reliability. In 2015, InfiniBand\u2019s share in the TOP500 supercomputer list exceeded 50%, making it the preferred internal connection technology for supercomputers.<\/p>\n\n\n\n<p>Currently, the primary supplier for InfiniBand (IB) architecture is Nvidia\u2019s Mellanox. The InfiniBand Trade Association (IBTA) was initially established by leading companies such as Intel, Microsoft, SUN, IBM, Compaq, and HP. In May 1999, Mellanox was founded in Israel by former employees from Intel and Galileo Technology. In 2001, they launched their first IB product. However, in 2002, Intel and Microsoft, the original giants in the IB camp, withdrew.<\/p>\n\n\n\n<p>In 2010, Mellanox merged with Voltaire, leaving Mellanox and QLogic as the main IB suppliers. In 2012, Intel re-entered the IB camp by acquiring QLogic\u2019s IB networking business for $125 million. Shortly after, Intel also purchased the \u201cGemini\u201d XT and \u201cAries\u201d XC supercomputing interconnect business from Cray for $140 million. They later developed the new Omni-Path interconnect technology based on IB and Aries.<\/p>\n\n\n\n<p>In 2013, Mellanox continued to expand by acquiring Kotura, a silicon photonics technology company, and IPtronics, a parallel optical interconnect chip manufacturer. By 2015, Mellanox held an 80% market share in the global InfiniBand market. In 2019, Nvidia successfully acquired Mellanox, outbidding competitors Intel and Microsoft with a $6.9 billion deal.<\/p>\n\n\n\n<p>Now, let\u2019s take a look at Nvidia\u2019s latest generation GB200 product:<\/p>\n\n\n\n<ol class=\"wp-block-list\" type=\"1\">\n<li>Internal Configuration of a Rack:<\/li>\n<\/ol>\n\n\n\n<p>Each rack contains 18 Compute trays (10 on top, 8 on the bottom) and 9 Switch trays.<\/p>\n\n\n\n<p>Copper Cable Cartridges connect Compute trays and Switch trays within the rack.<\/p>\n\n\n\n<p>Utilizing a liquid-cooling system, the GB200 achieves 25 times the performance compared to the H100\u2019s air-cooled infrastructure at the same power consumption.<\/p>\n\n\n\n<p>Compute Tray Components:<\/p>\n\n\n\n<p>Each individual Compute tray includes:<\/p>\n\n\n\n<p>2 GB200 Grace Blackwell Superchips<\/p>\n\n\n\n<p>4 ConnectX-800G InfiniBand Supernics<\/p>\n\n\n\n<p>1 BlueField-3 Data Processing Unit (DPU).<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/GB200-1.png\" alt=\"GB200\" class=\"wp-image-11831\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/GB200-1.png 938w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/GB200-1-300x196.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/GB200-1-768x502.png 768w\" sizes=\"(max-width: 938px) 100vw, 938px\" \/><\/figure>\n\n\n\n<p><strong>GB200 Grace Blackwell Superchip:<\/strong><strong><\/strong><\/p>\n\n\n\n<p>The GB200 Grace Blackwell Superchip comprises two Blackwell GPUs and one Grace CPU.<\/p>\n\n\n\n<p>Each individual Blackwell GPU is twice the size of the previous Hopper GPU architecture.<\/p>\n\n\n\n<p>However, its AI performance (FP4) is five times that of Hopper.<\/p>\n\n\n\n<p>Specifically, a single Blackwell GPU achieves approximately 20 petaFLOPS of AI performance (FP8).<\/p>\n\n\n\n<p>It features 8x 24GB HBM3e memory with an impressive 8TB\/s memory bandwidth.<\/p>\n\n\n\n<p>The GB200 has 18 NVLink ports, connecting to 18 NVLink switch chips, achieving bidirectional communication at 1.8TB\/s.<\/p>\n\n\n\n<p><strong>ConnectX-800G InfiniBand Supernic:<\/strong><strong><\/strong><\/p>\n\n\n\n<p>The ConnectX-800G InfiniBand Supernic enables end-to-end 800Gb\/s network connections and performance isolation.<\/p>\n\n\n\n<p>It is designed specifically for efficient management of multi-tenant AI clouds.<\/p>\n\n\n\n<p>Leveraging PCIe 6.0, it provides 800Gb\/s data throughput.<\/p>\n\n\n\n<p>The ConnectX-8 Supernic supports both single-port OSFP 224 and dual-port QSFP112 connectors.<\/p>\n\n\n\n<p>Additionally, it facilitates NVIDIA Socket Direct with 16-channel auxiliary card expansion.<\/p>\n\n\n\n<p><strong>Bluefield-3 DPU:<\/strong><strong><\/strong><\/p>\n\n\n\n<p>The Bluefield-3 DPU connects via 400Gb\/s Ethernet or NDR 400Gb\/s InfiniBand networks.<\/p>\n\n\n\n<p>It offloads, accelerates, and isolates software-defined networking, storage, security, and management functions.<\/p>\n\n\n\n<p>As a result, it significantly enhances data center performance, efficiency, and security.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/computer-tray-1-1024x558.png\" alt=\"computer tray\" class=\"wp-image-11832\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/computer-tray-1-1024x558.png 1024w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/computer-tray-1-300x163.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/computer-tray-1-768x418.png 768w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/computer-tray-1.png 1041w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Each Switch tray contains two NVLink switch chips.<\/p>\n\n\n\n<p>Each individual switch chip supports four interfaces, with a single interface achieving a 1.8TB\/s transfer rate.<\/p>\n\n\n\n<p>A Switch tray provides a total of 144 NVLink ports (100GB), resulting in 14.4TB\/s overall bandwidth.<\/p>\n\n\n\n<p>With nine Switch trays, you can fully connect 72 Blackwell GPUs, each with 18 NVLink ports (totaling 1296 NVLink ports).<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png\" alt=\"Switch tray\" class=\"wp-image-11833\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png 985w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1-300x148.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1-768x380.png 768w\" sizes=\"(max-width: 985px) 100vw, 985px\" \/><\/figure>\n\n\n\n<p>For internal cabinet connections:<\/p>\n\n\n\n<p>Compute trays and Switch trays are linked via fifth-generation NVLink.<\/p>\n\n\n\n<p>The bidirectional bandwidth of fifth-generation NVLink is 1.8TB\/s, twice that of the previous generation and over 14 times PCIe Gen5 bandwidth.<\/p>\n\n\n\n<p>This 1.8TB\/s GPU-to-GPU communication enables GPU expansion in AI and high-performance computing.<\/p>\n\n\n\n<p>Within the Compute tray, Superchip internal GPUs and CPUs are connected via NVLink Chip-to-Chip communication (bidirectional bandwidth of 900GB\/s).<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/nvlink.png\" alt=\"nvlink\" class=\"wp-image-11834\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/nvlink.png 586w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/nvlink-300x219.png 300w\" sizes=\"(max-width: 586px) 100vw, 586px\" \/><\/figure>\n\n\n\n<p>Intra-Cabinet Connections (Within a Single Rack):<\/p>\n\n\n\n<p>For configurations with up to 72 GPUs, the recommended solution within a single rack is to use copper cables (NVLink). The GB200\u2019s improved chip density and efficient liquid cooling allow for more GPUs to be deployed in a smaller space, making copper cables a cost-effective choice for intra-cabinet connections. However, long-distance transmission losses remain a concern for future iterations.<\/p>\n\n\n\n<p>When the number of GPUs exceeds 72, a single-layer network is insufficient. Upgrading to a higher-layer network structure is necessary. Two options are available: single NVLink and InfiniBand (IB) networking.<\/p>\n\n\n\n<p>Single NVLink Configuration:<\/p>\n\n\n\n<p>When connecting more than 72 but fewer than 576 GPUs, consider using an all-NVLink cluster architecture. The GPU-to-optical module ratio is 1:9.<\/p>\n\n\n\n<p>For scalability beyond a single rack, a recommended approach is the dual-rack NVL72 configuration. Each rack contains 18 Compute Trays and 9 Switch Trays. Notably, in the dual-rack version, each Compute Tray features only one Grace Blackwell Superchip (2 Blackwell GPUs + 1 Grace CPU). The content of the Switch Tray remains consistent across both single and dual-rack versions.<\/p>\n\n\n\n<p>With 36 Blackwell GPUs fully connected to 18 NVSwitch chips (totaling 648 ports), a 576-GPU cluster spans 16 dual-row cabinets. This results in a cumulative requirement of 10,368 ports, with a single-direction speed of 50GB\/s (bidirectional 100GB\/s). Assuming L1 to L2 network layers use 1.6T optical modules (200GB\/s), we need 5,184 1.6T optical modules. The GPU-to-optical module ratio is 1:9.<\/p>\n\n\n\n<p>InfiniBand (IB) Configuration:<\/p>\n\n\n\n<p>When the desired GPU count exceeds 72, consider IB networking. Using the latest NVIDIA Quantum-X800 Q3400 switch, the number of ports determines the maximum GPU capacity for different network layers.<\/p>\n\n\n\n<p>Compared to the NVIDIA Quantum-2 <a class=\"wpil_keyword_link\" href=\"https:\/\/www.fibermall.com\/blog\/qm9700.htm\"   title=\"QM9700\" data-wpil-keyword-link=\"linked\" target=\"_blank\">QM9700<\/a> with only 64 400G ports, the Quantum-X800 Q3400 offers 144 800G ports, allowing for a maximum of (144^2)\/2 = 10,368 interconnected GPUs.<\/p>\n\n\n\n<p>According to SemiAnalysis predictions, the GPU-to-1.6T optical module ratio is approximately 2.5 for a 2-layer network and around 3.5 for a 3-layer network.<\/p>\n<style>\r\n\r\n        .lwrp.link-whisper-related-posts{\r\n            \r\n            margin-top: 40px;\nmargin-bottom: 30px;\r\n        }\r\n        .lwrp .lwrp-title{\r\n            \r\n            \r\n        }\r\n        .lwrp .lwrp-description{\r\n            \r\n            \r\n\r\n        }\r\n        .lwrp .lwrp-list-container{\r\n        }\r\n        .lwrp .lwrp-list-multi-container{\r\n            display: flex;\r\n        }\r\n        .lwrp .lwrp-list-double{\r\n            width: 48%;\r\n        }\r\n        .lwrp .lwrp-list-triple{\r\n            width: 32%;\r\n        }\r\n        .lwrp .lwrp-list-row-container{\r\n            display: flex;\r\n            justify-content: space-between;\r\n        }\r\n        .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n            width: calc(100% - 20px);\r\n        }\r\n        .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n            \r\n            list-style: decimal;\r\n        }\r\n        .lwrp .lwrp-list-item img{\r\n            max-width: 100%;\r\n            height: auto;\r\n        }\r\n        .lwrp .lwrp-list-item.lwrp-empty-list-item{\r\n            background: initial !important;\r\n        }\r\n        .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n        .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n            \r\n                \r\n        }\r\n        @media screen and (max-width: 480px) {\r\n            .lwrp.link-whisper-related-posts{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-title{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-description{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-multi-container{\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-multi-container ul.lwrp-list{\r\n                margin-top: 0px;\r\n                margin-bottom: 0px;\r\n                padding-top: 0px;\r\n                padding-bottom: 0px;\r\n            }\r\n            .lwrp .lwrp-list-double,\r\n            .lwrp .lwrp-list-triple{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-row-container{\r\n                justify-content: initial;\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n            .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n                \r\n                    \r\n            }\r\n        }<\/style>\r\n<div id=\"link-whisper-related-posts-widget\" class=\"link-whisper-related-posts lwrp\">\r\n            <h3 class=\"lwrp-title\">Related Posts<\/h3>    \r\n        <div class=\"lwrp-list-container\">\r\n                                            <ul class=\"lwrp-list lwrp-list-single\">\r\n                    <li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/sfp28-nic.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Unveiling the Power of SFP28 NIC: Your Ultimate Guide to Network Interface Card<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/video\/40gbase-qsfp-univ-optical-transceiver.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">How to use MMF&#038;SMF cable to connect 40GBase QSFP+ UNIV Optical Transceiver?<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/calix-sfp.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Understanding the Calix\u00ae SFP Transceiver: An In-Depth Guide<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/800g-optical-module-fr-scenario.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">800G Optical Module &#8211; 4x200G Technical Analysis in FR Scenario<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/development-of-fiber-optic-transceivers.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Overview of the Development of Fiber Optic Transceivers<\/span><\/a><\/li>                <\/ul>\r\n                        <\/div>\r\n<\/div>","protected":false},"excerpt":{"rendered":"<p>The effective computational power of a cluster can be broken down into GPU utilization and the cluster\u2019s linear speedup. GPU utilization is influenced by factors such as chip architecture, process technology, memory, I\/O bottlenecks, inter-card interconnect bandwidth, topology, and power consumption. On the other hand, \u2018cluster linear speedup\u2019 depends on node communication capabilities, parallel training [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":11833,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":"","_wpscppro_dont_share_socialmedia":false,"_wpscppro_custom_social_share_image":0,"_facebook_share_type":"default","_twitter_share_type":"default","_linkedin_share_type":"default","_pinterest_share_type":"default","_linkedin_share_type_page":"default","_instagram_share_type":"default","_medium_share_type":"","_threads_share_type":"","_google_business_share_type":"","_selected_social_profile":[{"id":"skM9ewvR8O","platform":"linkedin","platformKey":0,"name":"Jason Xue","type":"person","thumbnail_url":"https:\/\/media.licdn.com\/dms\/image\/C5603AQErPqKD0j6qBg\/profile-displayphoto-shrink_100_100\/0\/1599138392315?e=1723075200&v=beta&t=joEkh1OeKQ0F-QpAPv4xxQyBdGlHyccIQZauRSs6RvU","share_type":"default"}],"_wpsp_enable_custom_social_template":false,"_wpsp_social_scheduling":{"enabled":false,"datetime":null,"platforms":[],"status":"template_only","dateOption":"today","timeOption":"now","customDays":"","customHours":"","customDate":"","customTime":"","schedulingType":"absolute"},"_wpsp_active_default_template":true},"categories":[2,29],"tags":[],"class_list":["post-11818","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-networking"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v20.13 (Yoast SEO v25.8) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Unleashing Performance: The Power of InfiniBand | FiberMall<\/title>\n<meta name=\"description\" content=\"InfiniBand, with its early adoption of RDMA, offers native advantages such as low latency, high bandwidth, and reliability.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Unleashing Performance: The Power of InfiniBand\" \/>\n<meta property=\"og:description\" content=\"The effective computational power of a cluster can be broken down into GPU utilization and the cluster\u2019s linear speedup. GPU utilization is influenced by\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm\" \/>\n<meta property=\"og:site_name\" content=\"fibermall.com\" \/>\n<meta property=\"article:published_time\" content=\"2024-08-09T08:37:14+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-08-09T08:37:28+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"985\" \/>\n\t<meta property=\"og:image:height\" content=\"487\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Mia\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mia\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm\"},\"author\":{\"name\":\"Mia\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/56a37c7b0e1e8c61823cd6ef48e8dce3\"},\"headline\":\"Unleashing Performance: The Power of InfiniBand\",\"datePublished\":\"2024-08-09T08:37:14+00:00\",\"dateModified\":\"2024-08-09T08:37:28+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm\"},\"wordCount\":1594,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png\",\"articleSection\":[\"Blog\",\"Networking\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm\",\"url\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm\",\"name\":\"Unleashing Performance: The Power of InfiniBand | FiberMall\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png\",\"datePublished\":\"2024-08-09T08:37:14+00:00\",\"dateModified\":\"2024-08-09T08:37:28+00:00\",\"description\":\"InfiniBand, with its early adoption of RDMA, offers native advantages such as low latency, high bandwidth, and reliability.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png\",\"width\":985,\"height\":487},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fibermall.com\/blog.htm\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Unleashing Performance: The Power of InfiniBand\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"name\":\"fibermall.com\",\"description\":\"Optical Communication Expert\",\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\",\"name\":\"fibermall.com\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"width\":200,\"height\":67,\"caption\":\"fibermall.com\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/56a37c7b0e1e8c61823cd6ef48e8dce3\",\"name\":\"Mia\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/2e83831e57346d5d451c45e43aa96d14fc758d8f72f7304810e123b36585382b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/2e83831e57346d5d451c45e43aa96d14fc758d8f72f7304810e123b36585382b?s=96&d=mm&r=g\",\"caption\":\"Mia\"},\"description\":\"Optical Transmission Consultant\",\"sameAs\":[\"https:\/\/www.fibermall.com\/\"],\"url\":\"https:\/\/www.fibermall.com\/blog.htm?author=4\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Unleashing Performance: The Power of InfiniBand | FiberMall","description":"InfiniBand, with its early adoption of RDMA, offers native advantages such as low latency, high bandwidth, and reliability.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm","og_locale":"en_US","og_type":"article","og_title":"Unleashing Performance: The Power of InfiniBand","og_description":"The effective computational power of a cluster can be broken down into GPU utilization and the cluster\u2019s linear speedup. GPU utilization is influenced by","og_url":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm","og_site_name":"fibermall.com","article_published_time":"2024-08-09T08:37:14+00:00","article_modified_time":"2024-08-09T08:37:28+00:00","og_image":[{"width":985,"height":487,"url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png","type":"image\/png"}],"author":"Mia","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Mia","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#article","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm"},"author":{"name":"Mia","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/56a37c7b0e1e8c61823cd6ef48e8dce3"},"headline":"Unleashing Performance: The Power of InfiniBand","datePublished":"2024-08-09T08:37:14+00:00","dateModified":"2024-08-09T08:37:28+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm"},"wordCount":1594,"commentCount":0,"publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png","articleSection":["Blog","Networking"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm","url":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm","name":"Unleashing Performance: The Power of InfiniBand | FiberMall","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png","datePublished":"2024-08-09T08:37:14+00:00","dateModified":"2024-08-09T08:37:28+00:00","description":"InfiniBand, with its early adoption of RDMA, offers native advantages such as low latency, high bandwidth, and reliability.","breadcrumb":{"@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#primaryimage","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/08\/Switch-tray-1.png","width":985,"height":487},{"@type":"BreadcrumbList","@id":"https:\/\/www.fibermall.com\/blog\/power-of-infiniband.htm#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fibermall.com\/blog.htm"},{"@type":"ListItem","position":2,"name":"Unleashing Performance: The Power of InfiniBand"}]},{"@type":"WebSite","@id":"https:\/\/www.fibermall.com\/blog.htm\/#website","url":"https:\/\/www.fibermall.com\/blog.htm\/","name":"fibermall.com","description":"Optical Communication Expert","publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization","name":"fibermall.com","url":"https:\/\/www.fibermall.com\/blog.htm\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","width":200,"height":67,"caption":"fibermall.com"},"image":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/56a37c7b0e1e8c61823cd6ef48e8dce3","name":"Mia","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/2e83831e57346d5d451c45e43aa96d14fc758d8f72f7304810e123b36585382b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/2e83831e57346d5d451c45e43aa96d14fc758d8f72f7304810e123b36585382b?s=96&d=mm&r=g","caption":"Mia"},"description":"Optical Transmission Consultant","sameAs":["https:\/\/www.fibermall.com\/"],"url":"https:\/\/www.fibermall.com\/blog.htm?author=4"}]}},"_links":{"self":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/11818","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=11818"}],"version-history":[{"count":9,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/11818\/revisions"}],"predecessor-version":[{"id":13182,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/11818\/revisions\/13182"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/media\/11833"}],"wp:attachment":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=11818"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=11818"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=11818"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}