Multiple spiral patterns in the transitional disk of HD 100546

Context. Protoplanetary disks around young stars harbor many structures related to planetary formation. Of particular interest, spiral patterns were discovered among several of these disks and are expected to be the sign of gravitational instabilities leading to giant planet formation or gravitational perturbations caused by already existing planets. In this context, the star HD 100546 presents some specific characteristics with a complex gaseous and dusty disk that includes spirals, as well as a possible planet in formation. Aims. The objective of this study is to analyze high-contrast and high angular resolution images of this emblematic system to shed light on critical steps in planet formation. Methods. We retrieved archival images obtained at Gemini in the near IR (Ks band) with the instrument NICI and processed the data using an advanced high contrast imaging technique that takes advantage of the angular differential imaging. Results. These new images reveal the spiral pattern previously identified with Hubble Space Telescope (HST) with an unprecedented resolution, while the large-scale structure of the disk is mostly cancelled by the data processing. The single pattern to the southeast in HST images is now resolved into a multi-armed spiral pattern. Using two models of a gravitational perturber orbiting in a gaseous disk, we attempted to constrain the characteristics of this perturber, assuming that each spiral is independent, and drew qualitative conclusions. The non-detection of the northeast spiral pattern observed in HST allows putting a lower limit on the intensity ratio between the two sides of the disk, which if interpreted as forward scattering, yields a larger anisotropic scattering than is derived in the visible. Also, we find that the spirals are likely to be spatially resolved with a thickness of about 5-10 AU. Finally, we did not detect the candidate planet in formation recently discovered in the L-p band, with a mass upper limit of 16-18 M-J.