Fossil group origins II. Unveiling the formation of the brightest group galaxies through their scaling relations

Context. Fossil systems are galaxy associations dominated by a relatively isolated, bright elliptical galaxy, surrounded by a group of smaller galaxies lacking L* objects. Because of this extreme environment, fossil groups (FGs) are ideal laboratories for studying the mass assembly of brightest group galaxies (BGGs). Aims. We analyzed the near-infrared photometric and structural properties of a sample of 20 BGGs present in FGs to better understand their formation mechanisms. They represent the largest sample studied to date. Methods. K-s-band deep images were used to study the structural properties of our sample galaxies. Their surface-brightness distribution was fitted to a Sersic profile using the GASP2D algorithm. Then, the standard scaling relations were derived for the first time for these galaxies and compared with those of normal ellipticals and brightest cluster galaxies in non-fossil systems. Results. The BGGs presented in this study represent a subset of the most massive galaxies in the Universe. We find that their ellipticity profiles are continuously increasing with the galactocentric radius. Our fossil BCGs follow closely the fundamental plane described by normal ellipticals. However, they depart from both the log sigma(0) vs. log L-Ks and log r(e) vs. log L-Ks relations described by intermediate-mass ellipticals. This occurs in the sense that our BGGs have larger effective radii and smaller velocity dispersions than those predicted by these relations. We also find that more elliptical galaxies systematically deviate from the previous relations, while rounder objects do not. No similar correlation was found with the Sersic index. Conclusions. The derived scaling relations can be interpreted in terms of the formation scenario of the BGGs. Because our BGGs follow the fundamental plane tilt but have larger effective radii than expected for intermediate-mass ellipticals, we suggest that they only went through dissipational mergers in an early stage of their evolution and then assembled the bulk of their mass through subsequent dry mergers, contrary to previous claims that BGGs in FGs were mainly formed by the merging of gas-rich galaxies.