Fiber Nonlinearity Mitigation in Single Carrier 400 G and 800 G Nyquist-WDM Systems

Fiber Kerr nonlinearity limits the achievable transmission distance and channel capacity of high speed optical communication systems. In this paper, we study the performance of C-band single carrier 400 G and 800 G 16-ary quadrature amplitude modulation (16 QAM) or 64-ary quadrature amplitude modulation (64 QAM) Nyquist wavelength division multiplexed (WDM) systems with emphasis on fiber nonlinearity and its mitigation. The nonlinearity mitigation methods of digital back-propagation (DBP) and coupled-equation digital back-propagation (CE-DBP) are discussed for single channel and multi-channel systems, respectively. The DBP algorithm is a general approach suitable for compensating intra-channel nonlinearity. The CE-DBP is an extension of the DBP, which can deal with both intra- and inter-channel nonlinearity by solving the coupled multi-channel Manakov equations. Based on the first order perturbation theory, we also analyze the application of Kalman filter (KF) to nonlinearity compensation, which utilizes correlation between adjacent symbols to compensate for Kerr nonlinearity. With numerical simulations, the performance of three nonlinearity equalization schemes of CE-DBP, DBP + KF, and CE-DBP + KF are compared. For nonlinearity mitigation in single carrier 400 G and 800 G WDM systems, the DBP + KF is more efficient than the CE-DBP considering five coupled channels. The CE-DBP + KF method has slightly better performance than the DBP + KF. However, the CE-DBP is quite complex due to multi-channel Manakov equation calculation, which demands broad-band joint detection with a single coherent receiver or several individual coherent receivers with time synchronization. In contrast, the DBP + KF only need the information of the target channel. We thus conclude that the DBP + KF is an attractive solution to mitigate fiber nonlinearity for single carrier 400 G and 800 G Nyquist-WDM systems. With the combination of the DBP and the KF, the transmission distance extensions are achieved as 12.5%, 14.3%, 22.6% and 28% for 400 G_64 QAM, 800 G_64 QAM, 400 G_16 QAM and 800 G_16 QAM WDM systems, respectively.