Reduced spin measurement back-action for a phase sensitivity ten times beyond the standard quantum limit

Fundamental quantum noise limits the precision of quantum-based detectors, for example limiting the ultimate precision of atomic clocks, which have applications in communication, navigation and tests of fundamental physics. Collective measurements of many quantum spins can project the ensemble into an entangled, spin-squeezed state with improved quantum-limited measurement resolution. However, measurement back-action has limited previous implementations of collective measurements to only modest observed enhancements in precision. Here, we experimentally demonstrate a collective measurement with reduced measurement back-action to produce and directly observe, with no background subtraction, a spin-squeezed state with phase resolution improved by a factor of 10.5(1.5) in variance, or 10.2(6) dB, compared to the initially unentangled ensemble of N=4.8x10(5) Rb-87 atoms. The measurement uses a cavity-enhanced probe of an optical cycling transition, mitigating back-action associated with state-changing transitions induced by the probe. This work establishes collective measurements as a powerful technique for generating useful entanglement for precision measurements.