Chernoff-Type Concentration of Empirical Probabilities in Relative Entropy

We study the relative entropy of the empirical probability vector with respect to the true probability vector in multinomial sampling of k categories, which, when multiplied by sample size n, is also the log-likelihood ratio statistic. We generalize a recent result and show that the moment generating function of the statistic is bounded by a polynomial of degree n on the unit interval, uniformly over all true probability vectors. We characterize the family of polynomials indexed by (k, n) and obtain explicit formulae. Consequently, we develop Chernoff-type tail bounds, including a closed-form version from a large sample expansion of the bound minimizer. Our bound dominates the classic method-of-types bound and is competitive with the state of the art. We demonstrate with an application to estimating the proportion of unseen butterflies.

Automated summary

The study investigates the relative entropy of the empirical probability vector with respect to the true probability vector in multinomial sampling, generalizing a recent result and showing the moment generating function of the statistic is bounded by a polynomial. By characterizing the family of polynomials and developing Chernoff-type tail bounds, including a closed-form version, the research demonstrates dominance over classic methods and competitiveness with the state of the art, as shown in an application to estimating the proportion of unseen butterflies.