Can a quantum nondemolition measurement improve the sensitivity of an atomic magnetometer?

We consider the limitations due to noise (e.g., quantum projection noise and photon shot-noise) on the sensitivity of an idealized atomic magnetometer that utilizes spin squeezing induced by a continuous quantum nondemolition measurement. Such a magnetometer measures spin precession of N atomic spins by detecting optical rotation of far-detuned light. We show that for very short measurement times, the optimal sensitivity scales as N-3/4; if strongly squeezed probe light is used, the Heisenberg limit of N-1 scaling can be achieved. However, if the measurement time exceeds tau(rel)/N-1/2 in the former case, or tau(rel)/N in the latter, where tau(rel) is the spin relaxation time, the scaling becomes N-1/2, as for a standard shot-noise-limited magnetometer.