Vacuum squeezing via polarization self-rotation and excess noise in hot Rb vapors

We present experimental and theoretical analysis of quantum fluctuation in a vacuum field in the presence of an orthogonal linearly polarized pump field propagating through a Rb vapor cell. Previously reported theoretical and experimental studies provided somewhat contradictory conclusions regarding the possibility of observing the 'squeezed vacuum' - the reduction of vacuum fluctuations below the standard quantum limit-in this system. Here, using the D1 transitions of Rb in a cell without a buffer as an example, we demonstrate that vacuum squeezing is corrupted by incoherent processes (such as spontaneous emission, elastic scattering, etc), and its observation is only possible in a specific small region of the experimental parameter space. Numerical simulations, in good agreement with the experiment, demonstrate that the two excited state hyperfine levels play a crucial role in the squeezing and excess noise production. The significant influence of far-detuned atoms on the field fluctuations at low noise frequencies imposes the explicit consideration of the full velocity distribution of the atomic vapor.