{"id":18271,"date":"2025-11-05T07:24:12","date_gmt":"2025-11-05T07:24:12","guid":{"rendered":"https:\/\/www.fibermall.com\/blog\/?p=18271"},"modified":"2026-07-17T09:19:02","modified_gmt":"2026-07-17T09:19:02","slug":"switching-nvidia-connectx-series-nics-from-infiniband-to-ethernet-mode-a-step-by-step-guide","status":"publish","type":"post","link":"https:\/\/www.fibermall.com\/blog\/nvidia-connectx-nic.htm","title":{"rendered":"Switching NVIDIA ConnectX Series NICs from InfiniBand to Ethernet Mode"},"content":{"rendered":"\n

A data center team installs an NVIDIA ConnectX adapter in an AI server and connects it to an Ethernet switch. Linux detects the PCIe device, the NVIDIA driver loads successfully, and the optical module is present\u2014but the Ethernet link remains down.<\/p>\n\n\n\n

In many cases, the problem is not the cable or switch. The ConnectX port is still configured for InfiniBand.<\/p>\n\n\n\n

Selected NVIDIA ConnectX Virtual Protocol Interconnect, or VPI, adapters can operate in either InfiniBand or Ethernet mode. This allows the same network card to support an InfiniBand AI fabric today and a RoCE v2 Ethernet network later without replacing the hardware.<\/p>\n\n\n\n

This flexibility is especially important for ConnectX-7<\/strong>\u00a0and ConnectX-8<\/strong>. ConnectX-7 is widely deployed in 400G NDR InfiniBand<\/a> and 400GbE networks, while ConnectX-8 targets 800G XDR InfiniBand and next-generation 800GbE AI infrastructure.<\/p>\n\n\n\n

The basic switching process is similar on both generations. However, ConnectX-8 may require additional attention to port layout, PCIe bandwidth, connector type, and 800G optics.<\/p>\n\n\n\n

This guide explains how to switch a supported ConnectX-7 or ConnectX-8 adapter from InfiniBand to Ethernet mode, configure the Linux interface, verify RoCE v2, and troubleshoot common 400G and 800G link problems.<\/p>\n\n\n\n

\n
\n

Table of Contents<\/p>\nToggle<\/span><\/path><\/svg><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n

  • ConnectX-7 vs ConnectX-8: Key Differences<\/a><\/li>
  • Confirm That the ConnectX Card Supports VPI<\/a><\/li>
  • Ethernet Mode Is Not the Same as RoCE v2<\/a><\/li>
  • Prerequisites<\/a><\/li>
  • Step 1: Identify the ConnectX Adapter<\/a><\/li>
  • Step 2: Start NVIDIA Firmware Tools<\/a><\/li>
  • Step 3: Query the Current Link Type<\/a><\/li>
  • Step 4: Back Up the Current Configuration<\/a><\/li>
  • Step 5: Switch ConnectX-7 to Ethernet Mode<\/a><\/li>
  • Step 6: Switch ConnectX-8 to Ethernet Mode<\/a>
    • ConnectX-8 C8180<\/a><\/li>
    • ConnectX-8 C8240<\/a><\/li><\/ul><\/li>
    • Step 7: Reboot or Power-Cycle the Server<\/a><\/li>
    • Step 8: Verify the Active Ethernet Link Layer<\/a><\/li>
    • Step 9: Identify and Configure the Ethernet Interface<\/a><\/li>
    • Step 10: Check Driver, Speed, and FEC<\/a><\/li>
    • ConnectX-7 Ethernet Compatibility<\/a><\/li>
    • ConnectX-8 Ethernet Compatibility<\/a><\/li>
    • Step 11: Verify RoCE v2<\/a><\/li>
    • Step 12: Test Ethernet and RDMA Connectivity<\/a><\/li>
    • Common Problems After Switching ConnectX Modes<\/a>
      • Ethernet Mode Is Active, but the Link Is Down<\/a><\/li>
      • LINK_TYPE Is Missing<\/a><\/li>
      • ConnectX-8 Runs at 400G Instead of 800G<\/a><\/li>
      • IP Traffic Works, but RoCE v2 Does Not<\/a><\/li><\/ul><\/li>
      • ConnectX-7 or ConnectX-8: Which Should You Use?<\/a><\/li>
      • Conclusion<\/a><\/li>
      • FAQ<\/a>
        • Can every ConnectX-7 and ConnectX-8 card switch between InfiniBand and Ethernet?<\/a><\/li>
        • What does LINK_TYPE_P1=2 mean?<\/a><\/li>
        • Can I use the same command on ConnectX-7 and ConnectX-8?<\/a><\/li>
        • Does switching ConnectX to Ethernet automatically enable RoCE v2?<\/a><\/li>
        • Does ConnectX-8 require PCIe Gen6?<\/a><\/li>
        • Can ConnectX-8 reuse ConnectX-7 optics?<\/a><\/li>
        • How do I verify Ethernet mode?<\/a><\/li>
        • Why is the ConnectX link still down after switching modes?<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n

          <\/span>ConnectX-7 vs ConnectX-8: Key Differences<\/strong><\/strong><\/span><\/h2>\n\n\n\n

          ConnectX-7 and ConnectX-8 <\/a>share the same general VPI architecture, but they are designed for different network generations.<\/p>\n\n\n\n

          Feature<\/strong><\/strong><\/td>ConnectX-7<\/strong><\/strong><\/td>ConnectX-8<\/strong><\/strong><\/td><\/tr>
          Maximum bandwidth<\/td>Up to 400Gb\/s<\/td>Up to 800Gb\/s<\/td><\/tr>
          InfiniBand generation<\/td>NDR<\/td>XDR<\/td><\/tr>
          Ethernet capability<\/td>Up to 400GbE<\/td>Up to 800GbE or dual 400GbE<\/td><\/tr>
          Host interface<\/td>PCIe Gen5 x16<\/td>PCIe Gen6-class connectivity<\/td><\/tr>
          Common NIC connector<\/td>OSFP or dual QSFP112<\/td>OSFP-RHS or dual QSFP112<\/td><\/tr>
          Typical GPU platforms<\/td>H100, H200, B200<\/td>B300, GB300 and newer platforms<\/td><\/tr>
          Configuration complexity<\/td>Usually a standard link-type change<\/td>May also require port-layout changes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

          For most ConnectX-7 VPI adapters, switching modes mainly involves changing LINK_TYPE_P1 or LINK_TYPE_P2 with mlxconfig, followed by a reboot.<\/p>\n\n\n\n

          ConnectX-8 follows the same general principle, but the exact process depends more heavily on the adapter model. Moving between one 800G port and two 400G ports may require additional port-splitting or physical-function settings.<\/p>\n\n\n\n

          \"ConnectX-7<\/figure>\n\n\n\n

          <\/span>Confirm That the ConnectX Card Supports VPI<\/strong><\/strong><\/span><\/h2>\n\n\n\n

          Not every product carrying the ConnectX name supports both InfiniBand and Ethernet.<\/p>\n\n\n\n

          Before changing any setting, verify:<\/p>\n\n\n\n

            \n
          • The full NVIDIA ordering part number<\/li>\n\n\n\n
          • Whether the adapter is a VPI model<\/li>\n\n\n\n
          • The number and type of physical ports<\/li>\n\n\n\n
          • Supported InfiniBand and Ethernet speeds<\/li>\n\n\n\n
          • Current firmware and PSID<\/li>\n\n\n\n
          • Available mlxconfig\u00a0parameters<\/li>\n\n\n\n
          • Required switch, cable, and transceiver configuration<\/li>\n<\/ul>\n\n\n\n

            A ConnectX family name alone is not enough. Some ConnectX-7 and ConnectX-8 variants may be Ethernet-only or use model-specific configurations.<\/p>\n\n\n\n

            If LINK_TYPE_P1 or LINK_TYPE_P2 does not appear in the mlxconfig output, the adapter may not support a conventional VPI mode change.<\/p>\n\n\n\n

            <\/span>Ethernet Mode Is Not the Same as RoCE v2<\/strong><\/strong><\/span><\/h2>\n\n\n\n

            Changing the ConnectX port to Ethernet mode changes the link layer. It allows the adapter to connect to an Ethernet switch and appear as a Linux Ethernet interface.<\/p>\n\n\n\n

            RoCE v2 is an RDMA transport that operates over UDP\/IP. A complete RoCE v2 deployment also requires:<\/p>\n\n\n\n

              \n
            • An IP address<\/li>\n\n\n\n
            • Valid RoCE v2 GID entries<\/li>\n\n\n\n
            • RDMA drivers and userspace tools<\/li>\n\n\n\n
            • Correct MTU<\/li>\n\n\n\n
            • Compatible VLAN and routing configuration<\/li>\n\n\n\n
            • Proper switch QoS<\/li>\n\n\n\n
            • Congestion control such as PFC and ECN<\/li>\n<\/ul>\n\n\n\n

              The firmware change enables Ethernet. The host and network configuration enable RoCE v2.<\/p>\n\n\n\n

              <\/span>Prerequisites<\/strong><\/strong><\/span><\/h2>\n\n\n\n

              Prepare the following before changing the ConnectX configuration:<\/p>\n\n\n\n

                \n
              • A supported ConnectX-7 or ConnectX-8 VPI adapter<\/li>\n\n\n\n
              • Root or sudo\u00a0access<\/li>\n\n\n\n
              • NVIDIA Firmware Tools, including mst\u00a0and mlxconfig\u00a0<\/li>\n\n\n\n
              • A compatible mlx5_core\u00a0driver<\/li>\n\n\n\n
              • RDMA utilities<\/li>\n\n\n\n
              • A maintenance window<\/li>\n\n\n\n
              • A compatible Ethernet switch port<\/li>\n\n\n\n
              • Correct optics, AOC, or DAC<\/li>\n\n\n\n
              • IP, VLAN, gateway, and MTU information<\/li>\n\n\n\n
              • A backup of the current configuration<\/li>\n<\/ul>\n\n\n\n

                The examples below use:<\/p>\n\n\n\n

                MST device: \/dev\/mst\/mt4129_pciconf0<\/p>\n\n\n\n

                RDMA device: mlx5_0<\/p>\n\n\n\n

                Ethernet interface: ens5f0np0<\/p>\n\n\n\n

                Port: 1<\/p>\n\n\n\n

                Replace these values with the names detected on your system.<\/p>\n\n\n\n

                \"Ethernet<\/figure>\n\n\n\n

                <\/span>Step 1: Identify the ConnectX Adapter<\/strong><\/strong><\/span><\/h2>\n\n\n\n

                Confirm that Linux detects the NIC:<\/p>\n\n\n\n

                lspci -nn | grep -i -E “Mellanox|NVIDIA”<\/p>\n\n\n\n

                Check the active driver:<\/p>\n\n\n\n

                lspci -k -s <PCI-address><\/p>\n\n\n\n

                Look for:<\/p>\n\n\n\n

                Kernel driver in use: mlx5_core<\/p>\n\n\n\n

                You can also check the loaded modules:<\/p>\n\n\n\n

                lsmod | grep mlx5<\/p>\n\n\n\n

                Typical modules include:<\/p>\n\n\n\n

                mlx5_core<\/p>\n\n\n\n

                mlx5_ib<\/p>\n\n\n\n

                Record the PCIe address, ConnectX generation, ordering part number, firmware version, and current interface names.<\/p>\n\n\n\n

                <\/span>Step 2: Start NVIDIA Firmware Tools<\/strong><\/strong><\/span><\/h2>\n\n\n\n

                Start the MST driver:<\/p>\n\n\n\n

                sudo mst start<\/p>\n\n\n\n

                Then display the detected devices:<\/p>\n\n\n\n

                sudo mst status -v<\/p>\n\n\n\n

                A ConnectX-7 device may appear similar to:<\/p>\n\n\n\n

                DEVICE_TYPE     MST                           PCI<\/p>\n\n\n\n

                ConnectX7       \/dev\/mst\/mt4129_pciconf0     5e:00.0<\/p>\n\n\n\n

                A ConnectX-8 card will use a different device identifier. Copy the exact \/dev\/mst\/ path for the next steps.<\/p>\n\n\n\n

                <\/span>Step 3: Query the Current Link Type<\/strong><\/strong><\/span><\/h2>\n\n\n\n

                Run:<\/p>\n\n\n\n

                sudo mlxconfig -d \/dev\/mst\/mt4129_pciconf0 query<\/p>\n\n\n\n

                To display the most relevant settings:<\/p>\n\n\n\n

                sudo mlxconfig -d \/dev\/mst\/mt4129_pciconf0 query \\<\/p>\n\n\n\n

                | grep -E “LINK_TYPE|SPLIT|PORT|NUM_OF_PF”<\/p>\n\n\n\n

                A typical ConnectX-7 VPI result may show:<\/p>\n\n\n\n

                LINK_TYPE_P1    IB(1)<\/p>\n\n\n\n

                LINK_TYPE_P2    IB(1)<\/p>\n\n\n\n

                The common values are:<\/p>\n\n\n\n

                  \n
                • 1\u00a0means InfiniBand.<\/li>\n\n\n\n
                • 2\u00a0means Ethernet.<\/li>\n<\/ul>\n\n\n\n

                  Another possible configuration is:<\/p>\n\n\n\n

                  LINK_TYPE_P1    ETH(2)<\/p>\n\n\n\n

                  LINK_TYPE_P2    IB(1)<\/p>\n\n\n\n

                  This means Port 1 is Ethernet and Port 2 is InfiniBand.<\/p>\n\n\n\n

                  For ConnectX-8, inspect the complete output. Additional parameters may control port splitting, physical functions, or the number of active links.<\/p>\n\n\n\n

                  <\/span>Step 4: Back Up the Current Configuration<\/strong><\/strong><\/span><\/h2>\n\n\n\n

                  Save the current firmware settings before making changes:<\/p>\n\n\n\n

                  sudo mlxconfig -d \/dev\/mst\/mt4129_pciconf0 query \\<\/p>\n\n\n\n

                  > connectx-config-before-change.txt<\/p>\n\n\n\n

                  Save the firmware information:<\/p>\n\n\n\n

                  sudo flint -d \/dev\/mst\/mt4129_pciconf0 query \\<\/p>\n\n\n\n

                  > connectx-firmware-before-change.txt<\/p>\n\n\n\n

                  Record the current network and RDMA interfaces:<\/p>\n\n\n\n

                  ip -br link > network-interfaces-before-change.txt<\/p>\n\n\n\n

                  rdma link show > rdma-links-before-change.txt<\/p>\n\n\n\n

                  This backup is useful if the interface names or port mappings change after rebooting.<\/p>\n\n\n\n

                  <\/span>Step 5: Switch ConnectX-7 to Ethernet Mode<\/strong><\/strong><\/span><\/h2>\n\n\n\n

                  To switch Port 1 of a supported ConnectX-7 VPI adapter to Ethernet:<\/p>\n\n\n\n

                  sudo mlxconfig -d \/dev\/mst\/mt4129_pciconf0 \\set LINK_TYPE_P1=2<\/p>\n\n\n\n

                  Confirm the change when prompted.<\/p>\n\n\n\n

                  To switch both ports to Ethernet:<\/p>\n\n\n\n

                  sudo mlxconfig -d \/dev\/mst\/mt4129_pciconf0 \\set LINK_TYPE_P1=2 LINK_TYPE_P2=2<\/p>\n\n\n\n

                  To keep Port 1 in Ethernet and Port 2 in InfiniBand:<\/p>\n\n\n\n

                  sudo mlxconfig -d \/dev\/mst\/mt4129_pciconf0 \\set LINK_TYPE_P1=2 LINK_TYPE_P2=1<\/p>\n\n\n\n

                  To restore Port 1 to InfiniBand:<\/p>\n\n\n\n

                  sudo mlxconfig -d \/dev\/mst\/mt4129_pciconf0 \\set LINK_TYPE_P1=1<\/p>\n\n\n\n

                  For most ConnectX-7 VPI deployments, changing the LINK_TYPE value is the main firmware modification required.<\/p>\n\n\n\n

                  \n