Congestion-control notification packets.<\/li>\n<\/ul>\n\n\n\nThese traffic types may have different requirements for latency, throughput, and packet-loss tolerance.<\/p>\n\n\n\n
A RoCE network therefore uses Quality of Service (QoS) classification to map traffic into appropriate switch priorities and hardware queues.<\/p>\n\n\n\n
Two commonly used classification mechanisms are:<\/p>\n\n\n\n
DSCP (Differentiated Services Code Point):<\/strong> A Layer 3 field in the IP header used to identify traffic classes across routed networks.<\/p>\n\n\n\nPCP (Priority Code Point):<\/strong> A three-bit priority field in an IEEE 802.1Q VLAN tag, commonly used for Layer 2 traffic prioritization.<\/p>\n\n\n\n<\/span>Example Traffic Classification Policy<\/strong><\/strong><\/span><\/h3>\n\n\n\nTraffic Type<\/strong><\/td>Example DSCP<\/strong><\/td>Example Priority<\/strong><\/td>Recommended Treatment<\/strong><\/td><\/tr>RoCE RDMA data<\/td> 24<\/td> 3<\/td> Dedicated lossless-capable class<\/td><\/tr> Congestion Notification Packets (CNP)<\/td> 48<\/td> 6<\/td> High-priority control traffic<\/td><\/tr> Regular TCP\/IP<\/td> 0<\/td> 0<\/td> Best-effort or separately managed class<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\nThese values illustrate one vendor-documented QoS design. They are not universal RoCE requirements and must be matched to the deployed NICs, switches, and QoS policies.<\/p>\n\n\n\n
<\/span>Why Consistent Classification Is Critical<\/strong><\/strong><\/span><\/h3>\n\n\n\nConsider a GPU server transmitting RoCE traffic with DSCP 24.<\/p>\n\n\n\n
The connected Leaf switch must correctly map that value to the intended RDMA queue. When the packet travels through a Spine switch and reaches another Leaf switch, the same QoS treatment must remain consistent.<\/p>\n\n\n\n
If one switch maps the RDMA traffic into a best-effort queue, the packet may no longer receive the intended buffering, congestion marking, or flow-control behavior.<\/p>\n\n\n\n
Therefore, end-to-end QoS consistency is a prerequisite for predictable RoCE network performance.<\/p>\n\n\n\n
<\/span>Switch Lossless Forwarding: PFC, Buffer Isolation, XOFF\/XON, and Headroom<\/strong><\/strong><\/span><\/h2>\n\n\n\nAfter RDMA packets enter the switch fabric, the forwarding hardware must absorb bursts and handle temporary congestion without allowing buffers to overflow unnecessarily.<\/p>\n\n\n\n
This is where Priority Flow Control, buffer management, and headroom allocation become essential.<\/p>\n\n\n\n
<\/span>Priority Flow Control (PFC)<\/strong><\/strong><\/span><\/h3>\n\n\n\nPFC, defined by IEEE 802.1Qbb, enables a switch to pause traffic on selected Ethernet priorities instead of stopping all traffic on a link.<\/p>\n\n\n\n
Unlike traditional Ethernet PAUSE, which affects the entire link, PFC provides finer-grained flow control.<\/p>\n\n\n\n
For example, if RDMA traffic uses priority 3, the switch can pause that priority while allowing other priorities to continue, subject to available resources and scheduling.<\/p>\n\n\n\n
<\/span>How PFC Works<\/strong><\/strong><\/span><\/h3>\n\n\n\nFigure 2. A congested switch uses PFC to temporarily stop the upstream sender on the affected priority, helping protect ingress buffers from overflow.<\/em><\/figcaption><\/figure>\n\n\n\nThe basic sequence is:<\/p>\n\n\n\n
\nRDMA traffic enters a switch queue or priority group.<\/li>\n\n\n\n Buffer occupancy increases as congestion develops.<\/li>\n\n\n\n When a configured threshold is reached, the switch sends a PFC pause frame to its directly connected upstream neighbor.<\/li>\n\n\n\n The upstream device temporarily stops transmitting traffic on the indicated priority.<\/li>\n\n\n\n When buffer occupancy has sufficiently decreased, transmission can resume.<\/li>\n<\/ol>\n\n\n\nBecause PFC operates between adjacent Ethernet devices, it does not directly regulate the original RDMA sender across an entire multi-hop fabric.<\/p>\n\n\n\n
<\/span>XOFF and XON Thresholds<\/strong><\/strong><\/span><\/h3>\n\n\n\nTwo concepts are commonly used to describe PFC buffer-control behavior:<\/p>\n\n\n\n
XOFF Threshold:<\/strong> The buffer occupancy level at which a switch triggers PFC to pause traffic on the affected priority.<\/p>\n\n\n\nXON Threshold:<\/strong> The lower occupancy level at which the switch allows paused traffic to resume, commonly by transmitting a zero-quanta PFC frame or through expiration of the pause timer.<\/p>\n\n\n\nThe difference between these thresholds provides hysteresis, helping prevent rapid oscillation between pause and resume states.<\/p>\n\n\n\n
Correct threshold configuration is essential.<\/p>\n\n\n\n
If XOFF is triggered too late, packets already in flight may exhaust the remaining buffer space. If it is triggered too early, the network may experience unnecessary pauses and reduced utilization.<\/p>\n\n\n\n
<\/span>Buffer Isolation and Shared Buffer Management<\/strong><\/strong><\/span><\/h3>\n\n\n\nRoCE switches commonly use configurable buffer pools, priority groups, and traffic-class queues to manage different traffic types.<\/p>\n\n\n\n
For example, the network may allocate:<\/p>\n\n\n\n
\nA lossless-capable buffer pool for RDMA traffic.<\/li>\n\n\n\n Separate lossy or best-effort resources for conventional Ethernet traffic.<\/li>\n\n\n\n Appropriate queue scheduling for congestion-control packets.<\/li>\n<\/ul>\n\n\n\nBuffer isolation helps prevent unrelated traffic from consuming all the resources needed to support the RDMA class.<\/p>\n\n\n\n
However, dedicated buffer space does not eliminate congestion. Excessive bursts or sustained oversubscription can still overwhelm available resources.<\/p>\n\n\n\n
<\/span>Why Headroom Matters<\/strong><\/strong><\/span><\/h3>\n\n\n\nHeadroom is additional buffer capacity reserved to absorb packets that continue arriving after a PFC pause request is triggered.<\/p>\n\n\n\n
A PFC pause frame cannot stop upstream traffic instantaneously. Propagation delay, device processing time, and data already in flight must all be considered.<\/p>\n\n\n\n
A simplified engineering model is:<\/p>\n\n\n\n
H_min \u2248 R \u00d7 T_response + H_margin<\/p>\n\n\n\n
Where:<\/p>\n\n\n\n
\nH_min = Minimum estimated headroom in bytes.<\/li>\n\n\n\n R = Incoming traffic rate in bytes per second.<\/li>\n\n\n\n T_response = Effective time before the upstream flow is stopped.<\/li>\n\n\n\n H_margin = Additional allowance for packet size, bursts, and implementation-specific behavior.<\/li>\n<\/ul>\n\n\n\nActual headroom sizing depends on the switch architecture, cable distance, link speed, traffic patterns, and vendor-specific buffering model.<\/p>\n\n\n\n
At 400G or 800G, even very short reaction delays can translate into significant amounts of data in flight.<\/p>\n\n\n\n
<\/span>Limitations of PFC<\/strong><\/strong><\/span><\/h3>\n\n\n\nAlthough PFC helps protect RDMA traffic against packet loss, excessive use can introduce performance problems, including pause propagation, head-of-line blocking, and possible deadlock under unfavorable conditions.<\/p>\n\n\n\n
Therefore, a robust RoCE architecture should use PFC as a protective mechanism rather than its primary approach to controlling persistent congestion.<\/p>\n\n\n\n
The preferred engineering objective is to minimize unnecessary PFC activation through effective end-to-end congestion control.<\/p>\n\n\n\n
<\/span>Congestion Detection: ECN and DCQCN<\/strong><\/strong><\/span><\/h2>\n\n\n\nPFC protects against buffer overflow by temporarily stopping traffic on a specific priority. However, repeatedly pausing traffic can increase latency and reduce network efficiency.<\/p>\n\n\n\n
To manage congestion more effectively, RoCE networks commonly combine Explicit Congestion Notification (ECN) with Data Center Quantized Congestion Notification (DCQCN).<\/p>\n\n\n\n
Together, these mechanisms allow the network to provide congestion feedback and enable the transmitting RDMA NIC to adjust its sending rate.<\/p>\n\n\n\n
<\/span>How ECN Detects Congestion<\/strong><\/strong><\/span><\/h3>\n\n\n\nECN is an IP-layer mechanism that signals congestion without necessarily dropping packets.<\/p>\n\n\n\n
An ECN-capable RoCE switch monitors queue occupancy or other supported congestion metrics. When congestion exceeds configured marking thresholds, the switch can mark eligible packets with the ECN Congestion Experienced (CE) codepoint.<\/p>\n\n\n\n
Instead of immediately discarding packets, the switch forwards the marked packets toward their destination, provided adequate buffer space remains.<\/p>\n\n\n\n
This gives the endpoint an opportunity to react to congestion before it develops into severe queue buildup or packet loss.<\/p>\n\n\n\n
<\/span>How DCQCN Controls the Sending Rate<\/strong><\/strong><\/span><\/h3>\n\n\n\nDCQCN is a congestion-control algorithm commonly implemented in RDMA-capable NICs.<\/p>\n\n\n\n
It uses feedback from congestion-aware switches and receiving NICs to regulate the transmitting NIC’s sending rate.<\/p>\n\n\n\n
A simplified DCQCN feedback loop includes three functional roles:<\/p>\n\n\n\n
\nCongestion Point (CP): A switch detecting congestion and marking eligible RoCEv2 packets with ECN.<\/li>\n\n\n\n Notification Point (NP): The receiving RoCE NIC that detects ECN-marked traffic and generates Congestion Notification Packets (CNPs).<\/li>\n\n\n\n Reaction Point (RP): The transmitting RoCE NIC that adjusts its injection rate after receiving CNP feedback.<\/li>\n<\/ul>\n\n\n\n<\/span>ECN + DCQCN Workflow<\/strong><\/strong><\/span><\/h3>\n\n\n\n