A telescope-ready approach for modal compensation of pyramid wavefront sensor optical gain

The pyramid wavefront sensor (PWFS) is the currently preferred design for high-sensitivity adaptive optics (AO) systems for extremely large telescopes (ELTs). Yet, nonlinearities of the signal retrieved from the PWFS pose a significant problem for achieving the full correction potential using this sensor, a problem that will only worsen with the increasing dimension of telescopes. This paper investigates the so-called optical gain (OG) phenomenon, a sensitivity reduction and an overall modification of the sensor response induced by the residual wavefront itself, with considerable effects in standard observation conditions for ELT-sized AO systems. Through extensive numerical analysis, this work proposes a formalism to measure and minimize the first-order nonlinearity error caused by optical gain variation, which uses a modal compensation technique of the calibrated reconstructor; this enables a notable increase in performance in faint guide stars or important seeing scenarios, for example from 16 to 30% H-band Strehl ratio for a sixteenth magnitude star in r(0) = 13 cm turbulence. Beyond the performance demonstrated by this compensation, a complete algorithm for realistic operation conditions is designed, which from dithering a few deformable mirror modes retrieves the optimal gains and updates the command matrix accordingly. The performance of this self-updating technique - which successfully allows automatic OG compensation regardless of the turbulent conditions, and its minimal interference with the scientific instrument are demonstrated through extensive end-to-end numerical simulations, all at the scale of an ELT instrument single-conjugate AO system.