Photometric selection of z ∼ 5 Lyman break galaxies in the ESO Remote Galaxy Survey

We describe the selection of a sample of photometrically defined Lyman break galaxies (LBGs) at z similar to 5 using the multiwavelength imaging data of the ESO (European Southern Observatory) Remote Galaxy Survey. The data are drawn from 10 widely separated fields covering a total sky area of 275 arcmin2. Starting with a simple colour (R - I > 1.3) and magnitude (I < 26.3) cut to isolate the Lyman break and then refining the sample by applying further optical and near-infrared photometric criteria we identify a sample of 253 LBG candidates. We carefully model the completeness of this sample and the factors that affect its reliability. There is considerable overlap between this sample and a spectroscopically confirmed sample drawn from the same survey and this allows us to determine the reliability of the optical photometric selection (similar to 60 per cent) and to show that the reliability can be significantly improved (to similar to 80 per cent) by applying near-infrared waveband criteria to exclude very red contaminants. Even this high level of reliability may compromise some statistical studies of LBG properties. We show that over 30 per cent of the highest reliability candidates have multiple ultraviolet (UV) luminous components and/or disturbed morphology in Hubble Space Telescope imaging, though it is unclear whether this represents multiple interacting/merging sources or individual large sources with multiple UV bright regions. Using this sample we confirm that the normalization of the bright end of the z = 5 UV luminosity function (down to M*) is lower than the same at z = 4 by a factor of 3. Using a Schechter fit we determine M*(UV) = -20.9 +/- 0.2. We discuss whether it is reasonable to expect the UV luminosity function to follow a Schechter function, given the UV emission is short lived and stochastic, and does not necessarily trace the underlying mass of the galaxy.