Spin-orbit physics of j=1/2 Mott insulators on the triangular lattice

The physics of spin-orbital entanglement in effective j = 1 2 Mott insulators, which have been experimentally observed for various 5d transition-metal oxides, has sparked an interest in Heisenberg-Kitaev (HK) models thought to capture their essential microscopic interactions. Here, we argue that the recently synthesized Ba3IrTi2O9 is a prime candidate for a microscopic realization of the triangular HK model, a conceptually interesting model for its interplay of geometric and exchange frustration. We establish that an infinitesimal Kitaev exchange destabilizes the 120 degrees. order of the quantum Heisenberg model. This results in the formation of an extended Z(2)-vortex crystal phase in the parameter regime most likely relevant to the real material, which can be experimentally identified with spherical neutron polarimetry. Moreover, using a combination of analytical and numerical techniques, we map out the entire phase diagram of the model, which further includes various ordered phases as well as an extended nematic phase around the antiferromagnetic Kitaev point.