An unbiased measurement of the UV background and its evolution via the proximity effect in quasar spectra

We investigated a set of high-resolution (R similar to 45 000), high signal-to-noise (S/N similar to 70) quasar spectra to search for the signature of the so-called proximity effect in the H I Ly alpha forest. The sample consists of 40 bright quasars with redshifts in the range 2.1 < z < 4.7. Using the flux transmission statistic, we determined the redshift evolution of the H I effective optical depth in the Lyman forest between 2 less than or similar to z less than or similar to 4.5, finding good agreement with previous measurements based on smaller samples. We also see the previously reported dip in tau(eff)(z) around redshift z similar to 3.3, but as the significance of that feature is only 2.6 sigma, we consider this detection tentative. Comparing the flux transmission near each quasar with what was expected from the overall trend of tau(eff)(z), we clearly detect the proximity effect not only in the combined quasar sample, but also towards each individual line of sight at high significance, albeit with varying strength. We quantify this strength using a simple prescription based on a fiducial value for the intensity of the metagalactic UV background (UVB) radiation field at 1 Ryd, multiplied by a free parameter that varies from QSO to QSO. The observed proximity effect strength distribution (PESD) is asymmetric, with an extended tail towards values corresponding to a weak effect. We demonstrate that this is not simply an effect of gravitational clustering around quasars, as the same asymmetry is already present in the PESD predicted for purely Poissonian variance in the absorption lines. We present the results of running the same analysis on simulated quasar spectra generated by a simple Monte-Carlo code. Comparing the simulated PESD with observations, we argue that the standard method of determining the UVB intensity J(v0) by averaging over several lines of sight is heavily biased towards high values of J(v0) because of the PESD asymmetry. Using instead the mode of the PESD provides an estimate of J(v0) that is unbiased with respect to his effect. For our sample we get a modal value for the UVB intensity of log J(v0) = -21.51 +/- 0.15 (in units of erg cm(-2) s(-1) Hz(-1) sr(-1)) for a median quasar redshift of 2.73. With J(v0) fixed we then corrected tau(eff) near each quasar for local ionisation and estimated the amount of excess H I absorption attributed to gravitational clustering. On scales of similar to 3 Mpc, only a small minority of quasars show substantial overdensities of up to a factor of a few in tau(eff); these are exactly the objects with the weakest proximity effect signatures. After removing those quasars residing in overdense regions, we redetermined the UVB intensity using a hybrid approach of sample averaging and statistical correction for the PESD asymmetry bias, arriving at log J(v0) = -21.46(-0.21)(+0.14). This is the most accurate measurement of J(v0) to date. We present a new diagnostic based on the shape and width of the PESD that strongly supports our conclusion that there is no systematic overdensity bias for the proximity effect. This additional diagnostic breaks the otherwise unavoidable degeneracy of the proximity effect between UVB and overdensity. We then applied our hybrid approach to estimate the redshift evolution of the UVB intensity and found tentative evidence of a mild decrease in log J(v0) with increasing redshift, by a factor of similar to 0.4 from z = 2 to z = 4. Our resultsare in excellent agreement with earlier predictions for the evolving UVB intensity, and they also agree well with other methods of estimating the UVB intensity. In particular, our measured UVB evolution is much slower than the change in quasar space densities between z = 4 and z = 2, supporting the notion of a substantial contribution of star-forming galaxies to the UVB at high redshift.