{"id":7189,"date":"2024-01-26T06:02:25","date_gmt":"2024-01-26T06:02:25","guid":{"rendered":"https:\/\/www.fibermall.com\/blog\/?p=7189"},"modified":"2024-03-11T03:58:05","modified_gmt":"2024-03-11T03:58:05","slug":"nvidia-ai-gpu-servers-pcie-vs-sxm","status":"publish","type":"post","link":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm","title":{"rendered":"NVIDIA AI GPU Servers: PCIe vs. SXM"},"content":{"rendered":"\n<p>Nvidia\u2019s GPU interconnect technology has two main types of memory slots: PCIe and SXM. These two interfaces have different functions and performance.<\/p>\n\n\n\n<p>PCIe interface is a widely used general protocol. Although it has comprehensive functions, it has a relatively slow transmission speed for GPU interconnect. Nevertheless, PCIe interface GPU cards can still communicate with CPU and other GPU cards in the server through PCIe slots. In addition, they can also exchange data with devices on external server nodes through network cards. If users want to improve the transmission speed of PCIe GPU cards, they can use NVLink bridge to achieve fast communication between GPU and CPU. However, it should be noted that this method usually only supports the connection between two GPU cards. That is to say, PCIe GPU cards usually need to appear in pairs, connected by NVLink Bridge, and transmit data through PCIe channel. It is worth noting that the latest PCIe standard has a limit of 128GB\/s in network bandwidth.<\/p>\n\n\n\n<p>In contrast, the SXM interface is designed specifically for high-performance <a href=\"https:\/\/www.fibermall.com\/blog\/how-many-optical-transceivers-needed-for-gpu.htm\" target=\"_blank\" rel=\"noreferrer noopener\">GPU interconnection<\/a>. It uses a dedicated protocol laid out on the circuit board, which enables SXM to offer higher transmission speed and better native NVLink support than PCIe in terms of inter-card connection. Its memory bandwidth is also superior to PCIe. SXM architecture is especially suitable for connecting GPUs to NVIDIA\u2019s proprietary DGX and HGX systems. In these systems, SXM-based GPUs are connected through NVSwitch integrated on the motherboard, without relying on PCIe for communication. This design allows SXM to support up to 8 GPUs interlinked, achieving very high bandwidth. For example, the uncut A100 and H100 can reach 600GB\/s and 900GB\/s of bandwidth respectively, while the slightly cut A800 and H800 can reach 400GB\/s of bandwidth.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-GPU-to-GPU-1024x350.png\" alt=\"PCIe GPU to GPU\" class=\"wp-image-7191\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-GPU-to-GPU-1024x350.png 1024w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-GPU-to-GPU-300x102.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-GPU-to-GPU-768x262.png 768w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-GPU-to-GPU-1536x525.png 1536w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-GPU-to-GPU.png 1982w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Users should generally weigh the pros and cons of PCIe-based or SXM-based GPUs according to their specific application scenarios and performance requirements.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"NVIDIA AI GPU Servers: PCIe vs. SXM | FiberMall\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/56aQgDpbUeQ?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_76 ez-toc-wrap-left counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\/#PCIe_PCI_Express_Introduction\" >PCIe (PCI Express) Introduction<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\/#PCIe_Throughput_Calculation\" >PCIe Throughput Calculation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\/#NVLink_A_high-speed_interconnect_technology_to_accelerate_GPU_communication\" >NVLink: A high-speed interconnect technology to accelerate GPU communication<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\/#NVSwitch_Seamless_High-Bandwidth_Multi-GPU_Communication\" >NVSwitch: Seamless, High-Bandwidth Multi-GPU Communication<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\/#Related_Posts\" >Related Posts<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\" id=\"h-pcie-pci-express-introduction\"><span class=\"ez-toc-section\" id=\"PCIe_PCI_Express_Introduction\"><\/span><strong>PCIe (PCI Express) Introduction <\/strong><strong><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>PCIe, short for PCI Express, is an advanced computer expansion bus standard, whose core goal is to improve data throughput and communication speed between devices. As a full-duplex connection bus, the data transfer rate of PCIe depends on its number of lanes. Each lane consists of two pairs of data lines (one for sending, one for receiving), and each pair of data lines contains two differential lines. For example, X1 means one lane and four data lines, which can transfer 1bit data per direction per clock cycle; while X2 means two lanes and eight data lines, which can transfer 2bit data per cycle. Similarly, there are also X12, X16, X32 and other configurations.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"652\" height=\"1024\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCI-Express-652x1024.png\" alt=\"PCI Express\" class=\"wp-image-7192\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCI-Express-652x1024.png 652w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCI-Express-191x300.png 191w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCI-Express.png 720w\" sizes=\"(max-width: 652px) 100vw, 652px\" \/><\/figure>\n\n\n\n<p>Since the release of PCIe 1.0 in 2003, its transmission rate has been increasing steadily. PCIe 1.0 supports a transmission rate of 250 MB\/s per channel, with a total transmission rate of 2.5 GT\/s. By 2007, the PCIe 2.0 specification doubled the total transmission rate to 5 GT\/s, and the transmission rate per channel also increased to 500 MB\/s. By 2022, the PCIe 6.0 specification further increased the total transmission rate to 64 GT\/s. In June of the same year, the PCI-SIG alliance announced the PCIe 7.0 specification, which is expected to achieve a single-channel (x1) unidirectional transmission rate of 128 GT\/s in the final version to be released in 2025. &nbsp;&nbsp;&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-large is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-bandwidth-1024x314.png\" alt=\"PCIe bandwidth\" class=\"wp-image-7193\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-bandwidth-1024x314.png 1024w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-bandwidth-300x92.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-bandwidth-768x235.png 768w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/PCIe-bandwidth.png 1080w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-pcie-throughput-calculation\"><span class=\"ez-toc-section\" id=\"PCIe_Throughput_Calculation\"><\/span><strong>PCIe Throughput Calculation <\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>When calculating PCIe throughput (or available bandwidth), two factors need to be considered: transmission rate and encoding scheme. The transmission rate is usually expressed in GT\/s (Giga Transitions per second), which describes the speed attribute of the physical layer communication protocol, rather than the number of bits transmitted per second (Gbps). This is because the transmission rate includes overhead bits that do not provide additional throughput. For example, PCIe 1.x and PCIe 2.x use the 8b\/10b encoding scheme, which means that 20% of the original channel bandwidth is used for overhead.<\/p>\n\n\n\n<p class=\"has-text-align-left\">Therefore, although the PCIe 2.0 protocol supports a transmission rate of 5.0 GT\/s, due to the use of the 8b\/10b encoding solution, the actual effective rate of each lane is 5*8\/10=4 Gbps, or 500 MB\/s. Similarly, the PCIe 3.0 protocol supports a transmission rate of 8.0 GT\/s, but after using the 128b\/130b encoding scheme, the actual effective rate of each lane is about 7.877 Gbps, or 984.6 MB\/s.<\/p>\n\n\n\n<p>PCIe architecture consists of PCIe devices of different types, such as Root Complex (RC), Switch, and Endpoint (EP). Among them, RC is the only one in the bus architecture, responsible for connecting the processor and memory subsystems with the I\/O devices. The function of the Switch is usually implemented by software, containing two or more logical PCI-to-PCI bridges (PCI-PCI Bridge) to ensure compatibility with existing PCI devices. This architecture provides strong support for high-performance computing and communication.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/endpoint.png\" alt=\"endpoint\" class=\"wp-image-7194\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/endpoint.png 852w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/endpoint-300x211.png 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/endpoint-768x541.png 768w\" sizes=\"(max-width: 852px) 100vw, 852px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-nvlink-a-high-speed-interconnect-technology-to-accelerate-gpu-communication\"><span class=\"ez-toc-section\" id=\"NVLink_A_high-speed_interconnect_technology_to_accelerate_GPU_communication\"><\/span><strong>NVLink: A high-speed interconnect technology to accelerate GPU communication<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>As the computing power continues to increase, multi-GPU cards become the key to improving performance. However, the traditional PCIe bus is prone to be a bottleneck in data transfer rate and latency, limiting the efficiency and performance of GPU parallel computing.<\/p>\n\n\n\n<p>To solve this problem, NVIDIA introduced GPUDirect P2P technology, which allows GPUs to directly access the memory of other GPUs through PCI Express, reducing the latency of data exchange. But even so, due to the PCI Express bus protocol and topology limitations, it still cannot meet the higher bandwidth requirements. For this reason, NVIDIA further launched NVLink high-speed interconnect technology. NVLink aims to speed up the data transfer between CPU and GPU, and GPU and GPU, providing a more efficient and low-latency interconnect solution. Since its first release in 2014, NVLink has been continuously upgraded, and the bandwidth has been increasing. From P100\u2019s 160GB\/s to V100\u2019s 300GB\/s, and then to A100\u2019s 600GB\/s, NVLink\u2019s bandwidth has been increasing, providing strong support for high-performance computing.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"ast-oembed-container \" style=\"height: 100%;\"><iframe title=\"How to Install a NVIDIA Nvlink 8-GPU Server? | FiberMall\" width=\"500\" height=\"281\" src=\"https:\/\/www.youtube.com\/embed\/74w-VrGGlpA?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<p>NVLink high-speed interconnect has two main implementation methods: one is through a bridge, and the other is to integrate NVLink interface on the motherboard. Both methods can achieve high-speed interconnection between GPUs and improve system performance.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png\" alt=\"nvidia nvswitch\" class=\"wp-image-7199\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png 688w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch-300x175.png 300w\" sizes=\"(max-width: 688px) 100vw, 688px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-nvswitch-seamless-high-bandwidth-multi-gpu-communication\"><span class=\"ez-toc-section\" id=\"NVSwitch_Seamless_High-Bandwidth_Multi-GPU_Communication\"><\/span><strong>NVSwitch: Seamless, High-Bandwidth Multi-GPU Communication<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>To solve the problem of imbalanced communication among GPUs, NVIDIA introduced the NVSwitch chip. NVSwitch is a physical chip (ASIC) similar to a switch, which can connect multiple GPUs at high speed through the NVLink interface. It can create a seamless, high-bandwidth multi-node GPU cluster, enabling all GPUs to work together in a cluster with full-bandwidth connectivity, thereby improving the communication efficiency and bandwidth among multiple GPUs within the server.<\/p>\n\n\n\n<p>The combination of NVLink and NVSwitch enables NVIDIA to efficiently scale AI performance to multiple GPUs, providing powerful support for high-performance computing and artificial intelligence applications. Since the release of the first generation of NVSwitch in 2018, NVSwitch has evolved to the third generation. The third generation of NVSwitch is built with TSMC\u2019s 4N process, and each chip has 64 NVLink 4.0 ports, with a communication rate of up to 900GB\/s between GPUs. This innovative technology provides a broader development space for future high-performance computing and artificial intelligence applications.<\/p>\n\n\n\n<figure class=\"wp-block-image aligncenter size-full is-resized\"><img decoding=\"async\" src=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvlink-performance-1.jpg\" alt=\"nvlink performance\" class=\"wp-image-7196\" style=\"width:800px\" width=\"800\" srcset=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvlink-performance-1.jpg 938w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvlink-performance-1-300x193.jpg 300w, https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvlink-performance-1-768x493.jpg 768w\" sizes=\"(max-width: 938px) 100vw, 938px\" \/><\/figure>\n<style>\r\n\r\n        .lwrp.link-whisper-related-posts{\r\n            \r\n            margin-top: 40px;\nmargin-bottom: 30px;\r\n        }\r\n        .lwrp .lwrp-title{\r\n            \r\n            \r\n        }\r\n        .lwrp .lwrp-description{\r\n            \r\n            \r\n\r\n        }\r\n        .lwrp .lwrp-list-container{\r\n        }\r\n        .lwrp .lwrp-list-multi-container{\r\n            display: flex;\r\n        }\r\n        .lwrp .lwrp-list-double{\r\n            width: 48%;\r\n        }\r\n        .lwrp .lwrp-list-triple{\r\n            width: 32%;\r\n        }\r\n        .lwrp .lwrp-list-row-container{\r\n            display: flex;\r\n            justify-content: space-between;\r\n        }\r\n        .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n            width: calc(100% - 20px);\r\n        }\r\n        .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n            \r\n            list-style: decimal;\r\n        }\r\n        .lwrp .lwrp-list-item img{\r\n            max-width: 100%;\r\n            height: auto;\r\n        }\r\n        .lwrp .lwrp-list-item.lwrp-empty-list-item{\r\n            background: initial !important;\r\n        }\r\n        .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n        .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n            \r\n                \r\n        }\r\n        @media screen and (max-width: 480px) {\r\n            .lwrp.link-whisper-related-posts{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-title{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-description{\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-multi-container{\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-multi-container ul.lwrp-list{\r\n                margin-top: 0px;\r\n                margin-bottom: 0px;\r\n                padding-top: 0px;\r\n                padding-bottom: 0px;\r\n            }\r\n            .lwrp .lwrp-list-double,\r\n            .lwrp .lwrp-list-triple{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-row-container{\r\n                justify-content: initial;\r\n                flex-direction: column;\r\n            }\r\n            .lwrp .lwrp-list-row-container .lwrp-list-item{\r\n                width: 100%;\r\n            }\r\n            .lwrp .lwrp-list-item:not(.lwrp-no-posts-message-item){\r\n                \r\n                \r\n            }\r\n            .lwrp .lwrp-list-item .lwrp-list-link .lwrp-list-link-title-text,\r\n            .lwrp .lwrp-list-item .lwrp-list-no-posts-message{\r\n                \r\n                    \r\n            }\r\n        }<\/style>\r\n<div id=\"link-whisper-related-posts-widget\" class=\"link-whisper-related-posts lwrp\">\r\n            <h3 class=\"lwrp-title\">Related Posts<\/h3>    \r\n        <div class=\"lwrp-list-container\">\r\n                                            <ul class=\"lwrp-list lwrp-list-single\">\r\n                    <li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/video\/the-use-of-sfp-10g-lr.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Design, construction and use of SFP 10G LR<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/400g-psm4.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Exploring the 400g PSM4 Optical Transceiver: A Comprehensive Guide<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/infiniband-cables.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">Understanding InfiniBand Cable: A Comprehensive Guide<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/infiniband-vs-roce.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">InfiniBand vs. RoCE Intelligent Computing Network<\/span><\/a><\/li><li class=\"lwrp-list-item\"><a href=\"https:\/\/www.fibermall.com\/blog\/dwdm-technology-fiber.htm\" class=\"lwrp-list-link\"><span class=\"lwrp-list-link-title-text\">DWDM Technology: Its Development and Application<\/span><\/a><\/li>                <\/ul>\r\n                        <\/div>\r\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Nvidia\u2019s GPU interconnect technology has two main types of memory slots: PCIe and SXM. These two interfaces have different functions and performance. PCIe interface is a widely used general protocol. Although it has comprehensive functions, it has a relatively slow transmission speed for GPU interconnect. Nevertheless, PCIe interface GPU cards can still communicate with CPU [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":7199,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":"","_wpscppro_dont_share_socialmedia":false,"_wpscppro_custom_social_share_image":0,"_facebook_share_type":"","_twitter_share_type":"","_linkedin_share_type":"","_pinterest_share_type":"","_linkedin_share_type_page":"","_instagram_share_type":"","_medium_share_type":"","_threads_share_type":"","_google_business_share_type":"","_selected_social_profile":[],"_wpsp_enable_custom_social_template":false,"_wpsp_social_scheduling":{"enabled":false,"datetime":null,"platforms":[],"status":"template_only","dateOption":"today","timeOption":"now","customDays":"","customHours":"","customDate":"","customTime":"","schedulingType":"absolute"},"_wpsp_active_default_template":true},"categories":[2,29],"tags":[],"class_list":["post-7189","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-networking"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v20.13 (Yoast SEO v25.8) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>NVIDIA AI GPU Servers: PCIe vs. SXM | FiberMall<\/title>\n<meta name=\"description\" content=\"Nvidia GPU interconnect technology has two main types of memory slots: PCIe and SXM. They have different functions and performance.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"NVIDIA AI GPU Servers: PCIe vs. SXM\" \/>\n<meta property=\"og:description\" content=\"Nvidia\u2019s GPU interconnect technology has two main types of memory slots: PCIe and SXM. These two interfaces have different functions and performance. PCIe\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\" \/>\n<meta property=\"og:site_name\" content=\"fibermall.com\" \/>\n<meta property=\"article:published_time\" content=\"2024-01-26T06:02:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-11T03:58:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png\" \/>\n\t<meta property=\"og:image:width\" content=\"688\" \/>\n\t<meta property=\"og:image:height\" content=\"401\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Ricky\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ricky\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\"},\"author\":{\"name\":\"Ricky\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/a8ba8a0499e3ebf85f6920e9a4f59f32\"},\"headline\":\"NVIDIA AI GPU Servers: PCIe vs. SXM\",\"datePublished\":\"2024-01-26T06:02:25+00:00\",\"dateModified\":\"2024-03-11T03:58:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\"},\"wordCount\":1172,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png\",\"articleSection\":[\"Blog\",\"Networking\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\",\"url\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\",\"name\":\"NVIDIA AI GPU Servers: PCIe vs. SXM | FiberMall\",\"isPartOf\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png\",\"datePublished\":\"2024-01-26T06:02:25+00:00\",\"dateModified\":\"2024-03-11T03:58:05+00:00\",\"description\":\"Nvidia GPU interconnect technology has two main types of memory slots: PCIe and SXM. They have different functions and performance.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png\",\"width\":688,\"height\":401,\"caption\":\"nvidia nvswitch\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fibermall.com\/blog.htm\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"NVIDIA AI GPU Servers: PCIe vs. SXM\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#website\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"name\":\"fibermall.com\",\"description\":\"Optical Communication Expert\",\"publisher\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#organization\",\"name\":\"fibermall.com\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"contentUrl\":\"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg\",\"width\":200,\"height\":67,\"caption\":\"fibermall.com\"},\"image\":{\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/a8ba8a0499e3ebf85f6920e9a4f59f32\",\"name\":\"Ricky\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4d7eb23fd15a1468d3ebd73b256dd17c8350e52200a5d5167902fe6c0a730a8d?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4d7eb23fd15a1468d3ebd73b256dd17c8350e52200a5d5167902fe6c0a730a8d?s=96&d=mm&r=g\",\"caption\":\"Ricky\"},\"description\":\"Optical Transmission Researcher, rich experience in solution design\",\"url\":\"https:\/\/www.fibermall.com\/blog.htm?author=3\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"NVIDIA AI GPU Servers: PCIe vs. SXM | FiberMall","description":"Nvidia GPU interconnect technology has two main types of memory slots: PCIe and SXM. They have different functions and performance.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm","og_locale":"en_US","og_type":"article","og_title":"NVIDIA AI GPU Servers: PCIe vs. SXM","og_description":"Nvidia\u2019s GPU interconnect technology has two main types of memory slots: PCIe and SXM. These two interfaces have different functions and performance. PCIe","og_url":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm","og_site_name":"fibermall.com","article_published_time":"2024-01-26T06:02:25+00:00","article_modified_time":"2024-03-11T03:58:05+00:00","og_image":[{"width":688,"height":401,"url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png","type":"image\/png"}],"author":"Ricky","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ricky","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#article","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm"},"author":{"name":"Ricky","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/a8ba8a0499e3ebf85f6920e9a4f59f32"},"headline":"NVIDIA AI GPU Servers: PCIe vs. SXM","datePublished":"2024-01-26T06:02:25+00:00","dateModified":"2024-03-11T03:58:05+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm"},"wordCount":1172,"commentCount":0,"publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png","articleSection":["Blog","Networking"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm","url":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm","name":"NVIDIA AI GPU Servers: PCIe vs. SXM | FiberMall","isPartOf":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage"},"image":{"@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage"},"thumbnailUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png","datePublished":"2024-01-26T06:02:25+00:00","dateModified":"2024-03-11T03:58:05+00:00","description":"Nvidia GPU interconnect technology has two main types of memory slots: PCIe and SXM. They have different functions and performance.","breadcrumb":{"@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#primaryimage","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/02\/nvidia-nvswitch.png","width":688,"height":401,"caption":"nvidia nvswitch"},{"@type":"BreadcrumbList","@id":"https:\/\/www.fibermall.com\/blog\/nvidia-ai-gpu-server-pcie-vs-sxm.htm#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fibermall.com\/blog.htm"},{"@type":"ListItem","position":2,"name":"NVIDIA AI GPU Servers: PCIe vs. SXM"}]},{"@type":"WebSite","@id":"https:\/\/www.fibermall.com\/blog.htm\/#website","url":"https:\/\/www.fibermall.com\/blog.htm\/","name":"fibermall.com","description":"Optical Communication Expert","publisher":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fibermall.com\/blog.htm\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.fibermall.com\/blog.htm\/#organization","name":"fibermall.com","url":"https:\/\/www.fibermall.com\/blog.htm\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/","url":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","contentUrl":"https:\/\/www.fibermall.com\/blog\/wp-content\/uploads\/2024\/12\/cropped-Fiber-Mall-Logo.jpg","width":200,"height":67,"caption":"fibermall.com"},"image":{"@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/a8ba8a0499e3ebf85f6920e9a4f59f32","name":"Ricky","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fibermall.com\/blog.htm\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/4d7eb23fd15a1468d3ebd73b256dd17c8350e52200a5d5167902fe6c0a730a8d?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4d7eb23fd15a1468d3ebd73b256dd17c8350e52200a5d5167902fe6c0a730a8d?s=96&d=mm&r=g","caption":"Ricky"},"description":"Optical Transmission Researcher, rich experience in solution design","url":"https:\/\/www.fibermall.com\/blog.htm?author=3"}]}},"_links":{"self":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/7189","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7189"}],"version-history":[{"count":7,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/7189\/revisions"}],"predecessor-version":[{"id":7369,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/posts\/7189\/revisions\/7369"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=\/wp\/v2\/media\/7199"}],"wp:attachment":[{"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7189"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7189"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fibermall.com\/blog.htm\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7189"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}