Joint intra and inter-channel nonlinearity compensation based on interpretable neural network for long-haul coherent systems

A novel joint intra and inter-channel nonlinearity compensation method is proposed, which is based on interpretable neural network (NN). For the first time, conventional cascaded digital back-propagation (DBP) and nonlinear polarization crosstalk canceller (NPCC) are deep unfolded into an NN architecture together based on their physical meanings. Verified by extensive simulations of 7-channel 20-GBaud DP-16QAM 3200-km coherent optical transmission, deep-unfolded DBP-NPCC (DU-DBP-NPCC) achieves 1 dB and 0.36 dB Q factor improvement at the launch power of -1 dBm/channel compared with chromatic dispersion compensation (CDC) and cascaded DBP-NPCC, respectively. Under the bit error rate threshold of 2 x 10(-2), DU-DBP-NPCC extends the maximum transmission reach by 28% (700 km) compared with CDC. Besides, 3 different training schemes of DU-DBP-NPCC are investigated, implying the effective signal-to-noise ratio is not the proper evaluation metric of nonlinearity compensation performance for DU-DBP-NPCC. Moreover, DU-DBP-NPCC costs 26% lower computational complexity compared with DBP-NPCC, providing a better choice for joint intra and inter-channel nonlinearity compensation in long-haul coherent systems. (C) 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement

Automated summary

This study introduces a novel joint intra and inter-channel nonlinearity compensation method based on interpretable neural network (NN) and evaluates its performance in coherent optical transmission systems through simulations. The results demonstrate significant advantages in transmission reach and Q factor improvement compared to conventional methods, with lower computational complexity.