Tests of statistical methods for estimating galaxy luminosity function and applications to the Hubble Deep Field

We study statistical methods for the estimation of the luminosity function (LF) of galaxies. We focus on four nonparametric estimators: the 1/V-max estimator, the stepwise maximum-likelihood estimator, Choloniewski's estimator, and the improved Lynden-Bell estimator. The performance of the 1/V-max estimator has recently been questioned, especially for the faint-end estimation of the LF. We improve these estimators for studies of the distant universe, and examine their performances for various classes of functional forms by Monte Carlo simulations. We also apply these estimation methods to the mock Two-Degree Field (2dF) redshift survey catalog prepared by Cole et al. We find that the 1/V-max estimator yields a completely unbiased result if there is no inhomogeneity, but is not robust against clusters or voids. This is consistent with the well-known results, and we do not confirm the bias trend of 1/V-max claimed by Willmer in the case of a homogeneous sample. We also find that the other three maximum-likelihood type estimators are quite robust and give consistent results with each other. In practice, we recommend Choloniewski's estimator for two reasons: (1) it simultaneously provides the shape and normalization of the LF; and (2) it is the fastest among these four estimators, because of its algorithmic simplicity. We then analyze the photometric redshift data of the Hubble Deep Field prepared by Fernandez-Soto et al, using the above four methods. We also derive the luminosity density, rho (L), at the B and I bands. Our B-band estimation is roughly consistent with that of Sawicki, Lin, & Yee, but it is a few times lower at 2.0 < z < 3.0. The evolution of rho (L)(I) is found to be less prominent.