Intra-Channel Nonlinearity Mitigation in Optical Fiber Transmission Systems Using Perturbation-Based Neural Network

In this work, a perturbation-based neural network (P-NN) scheme with an embedded bidirectional long short-term memory (biLSTM) layer is investigated to compensate for the Kerr fiber nonlinearity in optical fiber communication systems. Numerical simulations have been carried out in a 32-Gbaud dual-polarization 16-ary quadrature amplitude modulation (DP-16QAM) transmission system. It is shown that this P-NN equalizer can achieve signal-to-noise ratio improvements of similar to 1.37 dB and similar to 0.80 dB, compared to the use of a linear equalizer and a single step per span (StPS) digital back propagation (DBP) scheme, respectively. The P-NN equalizer requires lower computational complexity and can effectively compensate for intra-channel nonlinearity. Meanwhile, the performance of P-NN is more robust to the distortion caused by equalization enhanced phase noise (EEPN). Furthermore, it is also found that there exists a tradeoff between the choice of modulation format and the nonlinear equalization schemes for a given transmission distance.

Automated summary

This work investigates a perturbation-based neural network scheme with an embedded bidirectional long short-term memory layer to compensate for the Kerr fiber nonlinearity in optical fiber communication systems. Numerical simulations demonstrate the effectiveness and advantages of this scheme in high-speed transmission systems, achieving improvements in signal-to-noise ratio while reducing computational complexity and maintaining robustness to phase noise-induced distortion. Additionally, a tradeoff is found between the choice of modulation format and nonlinear equalization schemes for a given transmission distance.