Combined Neural Network and Adaptive DSP Training for Long-Haul Optical Communications

Machine Learning (ML) algorithms have shown to complement standard digital signal processing (DSP) tools in mitigating fiber nonlinearity and improving long-haul transmission performance. However, dynamic transmission impairments such as polarization effects and carrier phase noise corrupt the training data and conventional cost functions for neural network (NN) training become unsuitable. Simple cascade of ML and standard adaptive DSP blocks will also result in suboptimal transmission performance or require impractical training methodologies in presence of such dynamic transmission impairments. We show how the adaptive DSP blocks can be treated as extra stateful NN layers and be combined with the main NN so that standard backpropagation-like training algorithms in ML can be applied. In this case, the adaptive filters are viewed as NN states which are updated in the forward pass of the backpropagation. We study the combined training of linear and nonlinear parameters in the digital backpropagation (DBP) algorithm for fiber nonlinearity compensation (named generalized DBP (GDBP) hereafter), residual impairments, polarization effects, frequency offsets and carrier phase noise compensation filters as a single NN in a 7 x 288 Gb/s polarization multiplexed (PM)-16QAM transmission experiment over 1125 km. We derived the complete set of backpropagation-like gradients and state update equations for the static and dynamic parameters of the combined NN. We further proposed and open-sourced a JAX-based coding framework for their easy and practical implementation. GDBP is more generalized than other DBP variants proposed in literature and for a given total number of steps, GDBP is the first experimental demonstration of optimal single-channel DBP based-fiber nonlinearity compensation algorithm. In addition, for complexity constrained situations with shortened filter taps, GDBP enables a 1 dB performance improvement over other DBP variants.

Automated summary

This study explores the integration of adaptive DSP blocks as additional stateful NN layers, enabling the application of standard backpropagation-like training algorithms to address dynamic transmission impairments. By combining linear and nonlinear parameters in the digital backpropagation algorithm, a more generalized approach for fiber nonlinearity compensation is achieved, showcasing improved performance over existing DBP variants. Through the developed framework, GDBP demonstrates a novel and optimal solution for single-channel DBP-based fiber nonlinearity compensation, with potential performance gains in complexity-constrained scenarios.