Probing triple-Higgs productions via 4b2γ decay channel at a 100 TeV hadron collider

The quartic self-coupling of the Standard Model Higgs boson can only be measured by observing the triple-Higgs production process, but it is challenging for the LHC Run 2 or International Linear Collider (ILC) at a few TeV because of its extremely small production rate. In this paper, we present a detailed Monte Carlo simulation study of the triple-Higgs production through gluon fusion at a 100 TeV hadron collider and explore the feasibility of observing this production mode. We focus on the decay channel HHH -> b (b) over barb (b) over bar gamma gamma, investigating detector effects and optimizing the kinematic cuts to discriminate the signal from the backgrounds. Our study shows that, in order to observe the Standard Model triple-Higgs signal, the integrated luminosity of a 100 TeV hadron collider should be greater than 1.8 x 10(4) ab(-1). We also explore the dependence of the cross section upon the trilinear (lambda(3)) and quartic (lambda(4)) self-couplings of the Higgs. We find that, through a search in the triple-Higgs production, the parameters. 3 and. 4 can be restricted to the ranges [-1, 5] and [-20, 30], respectively. We also examine how new physics can change the production rate of triple-Higgs events. For example, in the singlet extension of the Standard Model, we find that the triple-Higgs production rate can be increased by a factor of O(10).