Fundamental performance limits of optical duobinary

We present a rigorous analysis defining the fundamental performance limits of duobinary line-coding for optical communications. First, we compare the back-to-back performance of duobinary and intensity modulation systems in an AWGN channel satisfying the Nyquist criterion, with both linear and quadratic receiver. We show that, also for duobinary and quadratic receiver, matched filtering is the best achievable condition. Then, we derive a detailed performance analysis of duobinary in an ASE-noise-limited direct-detection optical system considering noise on the entire space of polarizations. We show that for duobinary line-coding the expression of the bit error rate depends both on the shape of the transmitted pulse and on the receiver optical filter. Comparing duobinary coded and uncoded intensity modulation systems, we show the intrinsic advantages of using the duobinary line-coding in a system based on quadratic detection. Finally, some results for realistic setups are obtained through simulation and compared to the fundamental limits in order to show how close to those limits state-of-the-art systems can operate. (C) 2008 Optical Society of America