Planetary systems in close binary stars: the case of HD 196885 Combined astrometric and radial velocity study

Context. More than fifty candidate planets are presently known to orbit one component of a binary or a multiple star system. Planets can therefore form and survive in such an environment, although recent observing surveys indicate that short-separation binaries do not favour the presence of a planetary system around one of the components. Dynamical interactions with the secondary component can actually affect the giant planet formation and evolution significantly. For this reason, rare close binaries hosting giant planets offer an ideal laboratory for exploring the properties and the stability of such extreme planetary systems. Aims. In the course of our CFHT and VLT coronographic imaging survey dedicated to the search for faint companions of exoplanet host stars, a close (similar to 20 AU) secondary stellar companion to the exoplanet host HD 196885 A was discovered. In this study, our aim is to monitor the orbital motion of the binary companion. Combining radial velocity and high-contrast imaging observations, we aim to derive the orbital properties of the complete system and to test its dynamical stability to reveal its formation. Methods. For more than 4 years, we used the NaCo near-infrared adaptive optics instrument to monitor the astrometric position of HD 196885 B relative to A. The system was observed at five different epochs from August 2005 to August 2009 and accurate relative positions were determined. Results. Our observations fully reject the stationary background hypothesis for HD 196885 B. The two components are found to be comoving. The orbital motion of HD 196885 B is resolved well the orbital curvature is even detected. From our imaging data combined with published radial velocity measurements, we refine the complete orbital parameters of the stellar component. We derive for the first time its orbital inclination and its accurate mass. We also find solutions for the inner giant planet HD 196885 Ab that are compatible with previous independent radial velocity studies. Finally, we investigate the stability of the inner giant planet HD 196885 Ab with the binary companion proximity. Our dynamical simulations show that the system is currently and surprisingly more stable in a high mutual inclination configuration that falls in the Kozai resonance regime. If confirmed, this system would constitute one of the most compact non-coplanar systems known so far. It would raise several questions about its formation and stability.