Performance degradation only under concurrent workloads.<\/li>\n<\/ul>\n\n\n\nThe 30% figure is an illustrative troubleshooting scenario, not a measured NVIDIA benchmark.<\/p>\n\n\n\n
Key lesson:<\/strong> Ping verifies IP connectivity, not the congestion behavior of an RDMA fabric.<\/p>\n\n\n\n<\/span>One Lost Packet Costs an Order of Magnitude, Not Just a Little Latency<\/strong><\/strong><\/span><\/h3>\n\n\n\nRoCEv2 reliable transport uses acknowledgments and retransmission mechanisms to recover from packet loss.<\/p>\n\n\n\n
Depending on the NIC generation, transport behavior, and recovery implementation, a missing packet may trigger additional retransmission work, delay outstanding operations, and reduce effective throughput.<\/p>\n\n\n\n
Under heavy congestion, the resulting performance penalty can greatly exceed the transmission time of the lost packet.<\/p>\n\n\n\n
However, a tenfold penalty is not universal.<\/p>\n\n\n\n
Proper congestion management aims to prevent packet loss while avoiding excessive PFC pauses and head-of-line blocking.<\/p>\n\n\n\n
<\/span>NVIDIA AI Ethernet Product Stack: Switches, NICs, and DPUs<\/strong><\/strong><\/span><\/h2>\n\n\n\n<\/span>The Three-Layer Portfolio: Spectrum, ConnectX, BlueField<\/strong><\/strong><\/span><\/h3>\n\n\n\nNVIDIA’s Ethernet networking portfolio combines three major components:<\/p>\n\n\n\nComponent<\/strong><\/strong><\/td>Primary Function<\/strong><\/strong><\/td><\/tr>Spectrum Ethernet switches<\/td> High-bandwidth switching, routing, buffering, and congestion management<\/td><\/tr> ConnectX NICs and SuperNICs<\/td> RDMA connectivity, hardware acceleration, and GPU communication<\/td><\/tr> BlueField DPUs and SuperNICs<\/td> Infrastructure offload, security, storage networking, and accelerated connectivity<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nSpectrum-X integrates compatible switching, endpoint hardware, software, and congestion-control capabilities into an optimized Ethernet platform.<\/p>\n\n\n\n
A Spectrum-4 switch alone does not automatically provide the complete Spectrum-X feature set.<\/p>\n\n\n\n
<\/span>Switch Lineup: Spectrum-4 SN5600, SN5400, and Legacy Spectrum-3<\/strong><\/strong><\/span><\/h3>\n\n\n\nTwo important Spectrum-4 platforms are:<\/p>\n\n\n\nSwitch<\/strong><\/strong><\/td>Maximum Ethernet Configuration<\/strong><\/strong><\/td>Switching Capacity<\/strong><\/strong><\/td><\/tr>SN5400<\/td> 64 \u00d7 400GbE QSFP-DD<\/td> 25.6 Tbps<\/td><\/tr> SN5600<\/td> 64 \u00d7 800GbE OSFP<\/td> 51.2 Tbps<\/td><\/tr> Spectrum-3 systems<\/td> Model-dependent, typically up to 400GbE<\/td> Model-dependent<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThe SN5600 supports high-density 800G connections and flexible 400G breakout configurations.<\/p>\n\n\n\n
The SN5400 is suitable for 400G-based architectures requiring substantial port density.<\/p>\n\n\n\n
Spectrum-3 remains relevant for compatible existing deployments, although Spectrum-X capabilities and platform support must be verified separately.<\/p>\n\n\n\n
<\/span>Host-Side Adapters: ConnectX-7, ConnectX-8, and BlueField-3<\/strong><\/strong><\/span><\/h3>\n\n\n\nConnectX-7<\/strong> supports Ethernet connectivity up to 400Gb\/s, making it suitable for high-performance GPU servers.<\/p>\n\n\n\nConnectX-8 SuperNIC<\/strong> supports up to 800Gb\/s aggregate connectivity, including configurations using two 400G ports.<\/p>\n\n\n\nBlueField-3<\/strong> provides DPU-based infrastructure acceleration and, in SuperNIC variants, high-speed networking for AI workloads.<\/p>\n\n\n\nSelection should consider:<\/p>\n\n\n\n
\nRequired network bandwidth per GPU.<\/li>\n\n\n\n PCIe generation and available host bandwidth.<\/li>\n\n\n\n GPUDirect RDMA compatibility.<\/li>\n\n\n\n Ethernet port configuration.<\/li>\n\n\n\n Firmware and Spectrum-X software support.<\/li>\n<\/ul>\n\n\n\nNot every adapter variant supports the same capabilities.<\/p>\n\n\n\n