The discrepancy between dynamical and stellar masses in massive compact galaxies traces non-homology

For many massive compact galaxies, their dynamical masses (M-dyn proportional to Sigma(2)r(e)) are lower than their stellar masses (M-star). We analyse the unphysical mass discrepancy M-star/M-dyn > 1 on a stellar-mass-selected sample of early-type galaxies (M-star greater than or similar to 10(11) M-circle dot) at redshifts z similar to 0.2 to z similar to 1.1. We build stacked spectra for bins of redshift, size and stellar mass, obtain velocity dispersions, and infer dynamical masses using the virial relation M-dyn equivalent to K sigma(2)(e)r(e)/G with K = 5.0; this assumes homology between our galaxies and nearby massive ellipticals. Our sample is completed using literature data, including individual objects up to z similar to 2.5 and a large local reference sample from the Sloan Digital Sky Survey (SDSS). We find that, at all redshifts, the discrepancy between M-star and M-dyn grows as galaxies depart from the present-day relation between stellar mass and size: the more compact a galaxy, the larger its M-star/M-dyn. Current uncertainties in stellar masses cannot account for values of M-star/M-dyn above 1. Our results suggest that the homology hypothesis contained in the M-dyn formula above breaks down for compact galaxies. We provide an approximation to the virial coefficient K similar to 6.0[r(e)/(3.185 kpc)](-0.81)[M-star/(10(11) M-circle dot)](0.45), which solves the mass discrepancy problem. A rough approximation to the dynamical mass is given by M-dyn similar to [Sigma(e)/(200 km s(-1))](3.6)[r(e)/(3 kpc)](0.35)2.1 x 10(11) M-circle dot.