What Is a Coherent DSP? The Engine Inside Optical Transceivers

Priya runs the network operations center for a regional telecom operator. Last quarter her team deployed 400G ZR QSFP-DD modules between two data centers 95 km apart. Both link LEDs came up green on day one, and traffic passed. Everyone moved on to the next task.

Three weeks later, errors started climbing on one direction only. The fiber tested clean. The connectors were new. No module reported a single alarm.

What caught the problem was buried deep in the module’s telemetry, in a reading most engineers never look at: pre-FEC BER, rising steadily on the affected path. That reading exists because of the coherent DSP, the chip inside the module that makes 400G over 95 km possible in the first place. A coherent DSP is why a compact pluggable can reach across a metro area instead of stopping at a few hundred meters. It is also why those modules cost more, draw more power, and expose diagnostics an ordinary SFP never will.

This article explains what a coherent DSP is, what it does on transmit and receive, why it changed how networks are built, and how to read the telemetry it produces. If you are choosing between specific coherent modules right now, our complete guide to coherent optical modules maps the whole ZR family first, and this piece fills in the chip behind it.

What Is a Coherent DSP?

A coherent DSP is a digital signal processing ASIC inside a coherent optical transceiver. It prepares data for transmission and reconstructs it on receive, compensating fiber impairments and correcting errors so light modulated in amplitude, phase, and polarization can travel roughly 40 km to 10,000+ km at 100G to 1.2T+ per wavelength.

Put the DSP next to the optical engine and you get what the industry calls digital coherent optics (DCO). The optical engine supplies the physics: a tunable laser, a modulator, and a coherent receiver front end. The DSP supplies the intelligence. It decides which modulation format to use, cleans up what the fiber did to the signal, and fixes the errors that accumulated along the way.

That division of labor is worth holding onto. When you read that a module “has a coherent DSP” or “is DSP-based,” what it really means is that the module can undo fiber impairments electronically and adapt its own transmission mode in software. Neither is possible with direct-detect optics.

Why Coherent Optics Need a DSP (And SFP Links Don’t)

Direct-detect modules, the SFP and QSFP types carrying most data center traffic today, encode data in one dimension only: the intensity of the light. The transmitter turns the laser on and off, or shifts its brightness, and the receiver converts that brightness straight back to electrical signal. No local oscillator laser. No chip doing heavy math. For short links, that is plenty.

The problems start when distance grows. On a fiber run of 80 km, chromatic dispersion smears the light pulses, polarization drifts, and optical noise accumulates. An intensity-only signal has no way to sort out those impairments, because the receiver threw away the phase information at the first step. That is why IMDD links hit a practical wall around 10 km.

Coherent optics take a different path. The receiver mixes the incoming signal with a local oscillator laser, which recovers the full optical field: amplitude, phase, and both polarizations. That gives the receiver vastly more information, but also vastly more to process. The fiber has scrambled the phase and polarization in ways that change over time. Somewhere, that mess has to be untangled at billions of operations per second. That is the coherent DSP’s job.

So the rule is simple. Short reach, modest distance, and tight budgets point to direct detect. The moment reach, sensitivity, or per-wavelength capacity matters, you need a DSP to make sense of what arrives at the far end. Our coherent vs direct detection comparison walks through exactly where that line falls.

What the Coherent DSP Does on Transmit

The transmit side of the DSP builds the optical signal the same way a modem builds a signal for a phone line, just far faster and on light.

First, it maps bits into symbols. This is where the modulation format is set. DP-QPSK encodes 2 bits per symbol and is extremely robust, which is why it powers long-haul. DP-16QAM encodes 4 bits per symbol and doubles capacity, at the cost of less noise tolerance. Higher-order formats go further: 64QAM carries 6 bits per symbol but needs a very clean link.

Then the DSP converts those symbols into the analog waveforms that drive the modulator. It shapes the spectrum so the signal fits cleanly into a DWDM channel, and it pre-compensates known distortions before they ever reach the fiber.

One feature deserves special attention because it underpins the flexible-rate modules on the market today: probabilistic constellation shaping (PCS). Instead of using every constellation point equally, PCS uses lower-energy points more often, which measurably improves tolerance to noise. The result is a DSP that can trade reach for rate in software. That is how a single chip runs 100G QPSK on one link and 400G 16QAM on another without any hardware change. The 400G ZR vs ZR+ vs OpenZR+ breakdown shows how those software-selectable modes map to the standards you actually buy against.

What the Coherent DSP Does on Receive

The receive path is where the DSP earns its power budget.

When light arrives, the coherent front end mixes it with the local oscillator laser through a 90-degree optical hybrid and balanced photodetectors. That produces four electrical streams, one each for the in-phase and quadrature components of two polarizations. The DSP digitizes all four with high-speed analog-to-digital converters, then sets to work.

Three jobs dominate the receive processing.

First, chromatic dispersion compensation. Dispersion is deterministic and linear, so the DSP applies the mathematical inverse of the fiber’s dispersion as a digital filter. It can compensate arbitrary accumulated dispersion exactly, which is the reason optical dispersion-compensation hardware is gone from modern coherent networks.

Second, polarization recovery. PMD and polarization rotation are random and time-varying, so a fixed filter will not work. Instead, adaptive equalizers continuously estimate and invert the fiber’s effect on polarization. Algorithms like the constant modulus algorithm and decision-directed least mean squares do this in real time, tracking polarization changes far faster than they occur in real fiber.

Third, timing and carrier recovery. The DSP locks the symbol timing and corrects the frequency and phase offset between the transmitter laser and the local oscillator. Any drift here shows up immediately as errors.

Why Some Impairments Are Cheap and Others Are Not

Dispersion and PMD are linear impairments, so the DSP handles them with manageable math. Fiber nonlinearity, the optical Kerr effect, is a different story. Compensating it requires digital backpropagation that is far too power-hungry for a pluggable module, and standards like 400ZR and OpenZR+ simply omit it. In practice that leaves amplified spontaneous emission noise and nonlinearity as the two real limits on coherent reach, not dispersion. The FEC and power budgets that decide how much of each a module can tolerate are set by the standards themselves, which we get into next.

Forward Error Correction: How the DSP Extends Reach

Noise adds errors, and errors limit reach. Coherent systems do not retransmit dropped bits like a TCP session would. Instead, the transmitter adds parity overhead, and the receiver’s soft-decision FEC uses the statistical likelihood of each bit to correct errors locally. The coding gain from a good FEC is effectively free reach.

FEC choice is tightly coupled to the standard, and to power. The OIF 400ZR agreement uses CFEC, a concatenated FEC tuned for the low power budget of a ~15-18W module. OpenZR+ uses oFEC, a higher-gain soft-decision FEC that buys hundreds of extra kilometers of reach at the cost of roughly 3W more power. IEEE 100GBASE-ZR uses SCFEC for the 100G edge case.

Here is a scenario that makes the point. A colleague at a data center operator bought matched 400G ZR modules for two sites and watched the link refuse to pass traffic, even though both ends showed a carrier. The root cause was not the optics. One module was running the OIF 400ZR profile with CFEC and the other was running an OpenZR+ profile with oFEC. Two coherent DSPs, two FEC modes, no link. Once he set both ends to the same application code, the link came up instantly.

If you want the standards that govern these FEC modes in one place, the coherent optical module standards guide catalogs OIF 400ZR, OpenZR+, OpenROADM, and CMIS.

How the DSP Ended Dispersion Compensation Hardware

Before coherent DSPs reached pluggable form factors, engineers compensated dispersion in the optical domain. They installed dispersion compensation fiber, dispersion-compensation modules, and carefully planned where each sat in an amplified span. That hardware added loss, added nonlinearity, and forced operators to stock many compensator values for unknown link lengths.

Coherent reception changed the equation. Because dispersion is a well-understood, consistent effect, the DSP inverts it mathematically. The hardware disappeared. Laboratory and field systems have transmitted over 6,400 km of standard fiber with no in-line dispersion-compensation components at all.

The operational payoff is smaller than it sounds only because it is so large: link planning no longer revolves around dispersion budgets. Diego, a network architect we talked to, described pulling an entire shelf of dispersion-compensation modules out of a regional ring when he migrated the routes to coherent pluggables. “The fiber got simpler, the spares list got shorter, and the number of fiber types we had to stock dropped to one.” That cleanup is the DSP working invisibly in the background of every coherent deployment.

DSP Power and Cost: Why Coherent Modules Cost More

The DSP is the biggest power consumer in a coherent module, often close to half of the module’s total draw. That single fact explains a lot about form factors.

A 100G ZR QSFP28 runs roughly 5 to 6W, right at the ceiling of what a QSFP28 cage can dissipate, which is why 100G is the only coherent speed in that form factor. A 400G ZR QSFP-DD on a 7nm process draws about 15 to 18W. An 800G ZR OSFP module reaches around 24 to 25W, and an 800G ZR+ variant can climb to 26 to 32W, which is why those modules need the larger OSFP package. Our 400G ZR QSFP-DD buying guide and 100G ZR QSFP28 guide cover the practical power and thermal side of those form factors.

The power story has improved dramatically over a decade of CMOS scaling. Moving DSPs from older nodes to 7nm, then 5nm and 3nm, cut coherent module power by roughly 50x, which is exactly what made fitting a DSP into a pluggable possible. Per bit, the trend keeps improving: 800G runs near 0.038 W/Gbps versus about 0.059 W/Gbps for 400G.

Cost follows the silicon. A coherent DSP on a leading-edge node runs roughly 100 to 300 per module and can account for 20 to 40% of a module’s bill of materials. The local oscillator laser and precision optics add more on top. And because a leading-edge DSP design spin can cost $40 million or more, the supplier field stays small. That concentration shows up in the supply chain.

Coherent DSP Supply Chain: Who Makes the Chips

Two camps build coherent DSPs: merchant suppliers who sell chips to module makers, and vertically integrated vendors who design DSPs for their own systems.

On the merchant side, Marvell and Cisco’s Acacia subsidiary have led the pluggable market. Marvell’s Orion DSP brought 800G to pluggables on a 5nm process, and Cisco/Acacia’s Jannu DSP powers the CIM 8, the first single-carrier 1.2T pluggable. Coherent Corp, Retym, and Credo are pushing in with their own coherent and coherent-adjacent DSPs. On the in-house side, Ciena’s WaveLogic 6 family runs 1.6T metro and 800G over ultra-long-haul distances on a 3nm process, and Nokia combines its PSE-6s DSP with Infinera’s ICE7 line after the 2025 merger.

The market split tells you how fast this is moving. Cignal AI tracks coherent DSP supply by generation and by track, separating high-performance embedded DSPs from the compact pluggable parts. In 2025, 400ZR-class pluggables were roughly 93% of pluggable coherent shipments. That share is forecast to fall to about 61% in 2026 as 800G climbs toward 35%, with 800ZRx expected to ship more than 200,000 ports in 2026 and 1600ZRx arriving in 2027.

Procurement reality follows. Leading-edge coherent DSPs rely on 3nm and 2nm CMOS, and lead times have run past nine months. If your build is date-constrained, the DSP supply chain is not a background detail. Our 800G ZR and ZR+ guide flags the same warning from the module buyer’s side.

Coherent DSP Telemetry: Reading the Real Health of a Link

A green LED does not mean a healthy link. It means the link is up today. The DSP exposes how much margin that link actually has, and that is the reading Priya’s team needed.

Two values matter most. Pre-FEC BER is the bit error rate before error correction does its work. If it is low, the DSP is correcting almost nothing and the link has plenty of margin. If it climbs, the link is degrading, even though post-FEC errors are still zero. Q-margin and OSNR give related views of how far the signal sits from the failure point.

On Cisco gear, show controller coherentDSP exposes pre-FEC BER and OSNR directly. Arista surfaces similar coherent diagnostics through show interfaces transceiver and the CMIS coherent DSP/VDM registers. The workflow is consistent across vendors: establish a baseline when the link is new and healthy, then compare against it when something feels wrong.

Back to Priya. When she pulled the coherent DSP telemetry, she found pre-FEC BER climbing on the affected direction while the receive OSNR had dropped a few decibels. The physical layer still looked perfect. The culprit was an amplifier gain setting that had drifted after a maintenance window, quietly eating margin for weeks. No LED ever changed color. The DSP telemetry caught it before the link ever dropped a post-FEC packet.

For the full command-level troubleshooting workflow, the coherent module compatibility and troubleshooting guide covers CMIS power-state issues, unsupported-transceiver alarms, and how to verify a coherent link end to end.

Coherent DSP FAQ

What is a coherent DSP?

A coherent DSP is a digital signal processing ASIC inside a coherent optical transceiver. It builds the transmitted signal, reconstructs the received signal, compensates fiber impairments, and corrects errors so phase- and polarization-modulated light can travel long distances at very high rates.

Why do coherent optical modules need a DSP?

Coherent transmission encodes data on the amplitude, phase, and polarization of light. The fiber scrambles all three. Only a DSP can untangle that at speed, which is why every coherent pluggable contains one.

Does an SFP or DAC cable use a coherent DSP?

No. SFP optics and DAC cables are direct-detect or copper short-reach links. They carry intensity or electrical signals over short distances where impairments are small, so no coherent DSP is needed.

What does the DSP do on transmit versus receive?

On transmit it maps bits to symbols, shapes the spectrum, and pre-compensates known distortions. On receive it digitizes the signal, compensates chromatic dispersion and polarization effects, recovers timing and carrier phase, and decodes forward error correction.

How does the DSP compensate chromatic dispersion?

Dispersion is a deterministic linear effect. The DSP applies the inverse of the fiber’s dispersion as a digital filter, removing arbitrary accumulated dispersion in software. That is why dispersion-compensation modules are obsolete in coherent networks.

What is the difference between CFEC and oFEC?

CFEC is the lower-power concatenated FEC used by OIF 400ZR for links around 80 to 120 km. oFEC is the higher-gain soft-decision FEC used by OpenZR+ for longer regional and long-haul reach, at the cost of a few extra watts.

Which companies make coherent DSPs?

Marvell, Cisco/Acacia, Coherent Corp, Retym, and Credo sell merchant coherent DSPs. Ciena, Nokia, and Huawei design their own DSPs for their own optical systems.

How much power does a coherent DSP use?

Often close to half of a coherent module’s total power. A 100G ZR QSFP28 runs about 5 to 6W, a 400G ZR QSFP-DD about 15 to 18W, and an 800G ZR OSFP module around 24 to 32W depending on the ZR+ variant.

What is pre-FEC BER and why does it matter?

Pre-FEC BER is the bit error rate measured before forward error correction. It shows how much margin a link has. A rising pre-FEC BER warns of degradation while post-FEC errors are still zero, which makes it one of the earliest signs of trouble.

Conclusion

A coherent DSP is the reason modern optical modules can carry 100G to 1.6T over hundreds or thousands of kilometers. It builds the signal on transmit, reconstructs it on receive, corrects errors with FEC, and made dispersion-compensation hardware obsolete. Because it consumes much of a module’s power and cost, the DSP drives the price difference between a direct-detect optic and a coherent one. And because it decides reach and rate flexibility, understanding the DSP is really how you understand the module itself.

The practical takeaways are short. Match FEC modes at both ends of every coherent link. Budget for the DSP’s power draw when you pick a form factor. Treat DSP lead times as part of your procurement plan for 800G and beyond. And once a link is live, verify it with DSP telemetry, pre-FEC BER and Q-margin, not with the LED.

When you are ready to spec coherent modules for your network, we can help you match reach and rate to the right part. Explore our DWDM and WDM transceiver solutions, or talk to our networking engineers with your link distances and traffic plans.