Confronting predictions of the galaxy stellar mass function with observations at high redshift

We investigate the evolution of the galaxy stellar mass function at high redshift (z >= 5) using a pair of large cosmological hydrodynamical simulations: MassiveBlack and MassiveBlack-II. By combining these simulations, we can study the properties of galaxies with stellar masses greater than 10(8) M-circle dot h(-1) and (comoving) number densities of log(10)(phi [Mpc(-3) dex(-1) h(3)]) > -8. Observational determinations of the galaxy stellar mass function at very high redshift typically assume a relation between the observed ultraviolet (UV) luminosity and stellar mass-to-light ratio which is applied to high-redshift samples in order to estimate stellar masses. This relation can also be measured from the simulations. We do this, finding two significant differences with the usual observational assumption: it evolves strongly with redshift and has a different shape. Using this relation to make a consistent comparison between galaxy stellar mass functions, we find that at z=6 and above the simulation predictions are in good agreement with observed data over the whole mass range. Without using the correct UV luminosity and stellar mass-to-light ratio, the discrepancy would be up to two orders of magnitude for large galaxies (>10(10) M-circle dot h(-1)). At z = 5, however, the stellar mass function for low-mass galaxies (<10(9) M-circle dot h(-1)) is overpredicted by factors of a few, consistent with the behaviour of the UV luminosity function, and perhaps a sign that feedback in the simulation is not efficient enough for these galaxies.