Intelligent and adaptive intra-channel and inter-channel fiber nonlinearity compensation in coherent optical transmission systems

Fiber nonlinearity mitigation is a crucial technology for extending transmission reach and increasing channel capacity in high-baud rate wavelength division multiplexing (WDM) systems. In this work, we propose a novel, to the best of our knowledge, architecture that combines learned modified digital back-propagation (L-MDBP) to compensate for intra-channel nonlinearity and a two-stage decision directed least mean square (DDLMS) adaptive equalizer to mitigate inter-channel nonlinearity. By leveraging globally optimized model parameters and adaptive channel estimation, the proposed scheme achieves superior performance and lower computation complexity compared with conventional DBP. Specifically, in an 8 x 64 Gbaud 16-ary quadrature amplitude modulation (16QAM) experimental system over 1600 km of standard single-mode fiber (SSMF), our approach shows a 0.30-dB Q2-factor improvement and a complexity reduction of 82.3% compared with DBP with 8 steps per span (SPS). Furthermore, we enhance the adaptability of the architecture by introducing an online transfer learning (TL) technique, which requires only 2% of initial training epochs. & COPY; 2023 Optica Publishing Group

Automated summary

In this work, a novel architecture combining learned modified digital back-propagation (L-MDBP) and decision directed least mean square (DDLMS) adaptive equalizer is proposed to mitigate fiber nonlinearity in high-baud rate WDM systems. The proposed scheme achieves superior performance and lower computation complexity compared with conventional DBP by leveraging globally optimized model parameters and adaptive channel estimation. Furthermore, the adaptability of the architecture is enhanced using an online transfer learning (TL) technique with minimal initial training epochs.