Evidence for competing magnetic instabilities in underdoped YBa2Cu3O6+x

We report a polarized neutron-scattering study of the orbital-like magnetic order in strongly underdoped YBa2Cu3O6.45 and YBa2(Cu0.98Zn0.02)(3)O-6.6. Their hole-doping levels are located on both sides of the critical doping p(MI) of a metal-insulator transition inferred from transport measurements. Our study reveals a drop down of the orbital-like order slightly below p(MI) with a steep decrease of both the ordering temperature T-mag and the ordered moment. Above p(MI), substitution of quantum impurities does not change T-mag, whereas it lowers significantly the bulk ordered moment. The modifications of the orbital-like magnetic order are interpreted in terms of a competition with electronic liquid crystal phases around p(MI). This competition gives rise to a mixed magnetic state in YBa2Cu3O6.45 and a phase separation in YBa2(Cu0.98Zn0.02)(3)O-6.6.