Properties of z similar to 3-6 Lyman break galaxies I. Testing star formation histories and the SFR-mass relation with ALMA and near-IR spectroscopy

Context. The existence of a relation between the star formation rate (SFR) and stellar mass (M-*) of galaxies, and the evolution of the specific star formation rate (sSFR = SFR/M-*) are key questions for the understanding of early galaxy formation and evolution. Aims. We examine the dependence of derived physical parameters of distant Lyman break galaxies (LBGs) on the assumed star formation histories (SFHs) and age priors, their implications on the SFR-mass relation, and we propose observational tests to better constrain these quantities using IR observations and emission line studies. Methods. We use our spectral energy distribution (SED) fitting tool including the effects of nebular emission. We analyze a large sample of LBGs from redshift z similar to 3 to 6, assuming five different star formation histories, extending thereby our first analysis of this sample (de Barros et al. 2012, A& A, submitted, Paper II). In addition we predict the IR luminosities consistently with the SED fits, assuming that all radiation absorbed by dust is reemitted in the IR. Results. Compared to standard SED fits assuming constant SFR and neglecting nebular lines, models assuming variable SFHs yield systematically lower stellar masses, higher extinction, higher SFR, higher IR luminosities, and a wider range of equivalent widths for optical emission lines. Exponentially declining and delayed SFHs yield basically identical results. Exponentially rising SFHs with variable timescales yield similar masses, but somewhat higher extinction than exponentially declining ones. Distinguishing these SFHs from the currently available data is difficult. We find significant deviations between the derived SFR and IR luminosity from the commonly used SFR(IR) or SFR(IR+UV) calibration, due to differences in the SFHs and ages. For most LBGs at z greater than or similar to 3 we find that the standard calibrations will underestimate the true, current SFR derived from the SED fits, due to assumptions inconsistent with the SED models. Models with variable SFHs, favored statistically, yield generally a large scatter in the SFR-mass relation. We show the dependence of this scatter on assumptions of the SFH, the introduction of an age prior, and on the extinction law. We show that the true scatter in the SFR-mass relation can be significantly larger than inferred using SFR(UV) and/or SFR(IR), if the true star formation histories are variable and relatively young populations are present. We show that different SFHs, and hence differences in the derived SFR-mass relation and in the specific star formation rates, can be tested/constrained observationally with future IR observations with ALMA. Measurement of emission lines, such as Ha and [O II] lambda 3727, can also provide useful constraints on the SED models, and hence test the predicted physical parameters. Conclusions. Important results such as the finding of a large scatter in the SFR-mass relation at high-z and an increase of the specific star formation rate with redshift above z greater than or similar to 3 (cf. Paper II) can be tested observationally using IR observations and measurements of (rest-frame) optical emission lines. Consistent analysis of all the observables including the rest-frame UV to IR and emission lines are required to establish more precisely true SFR values and the scatter in the SFR-mass relation.