Unconventional pairing and electronic dimerization instabilities in the doped Kitaev-Heisenberg model

We study the quantum many-body instabilities of the t-J(K)-J(H) Kitaev-Heisenberg Hamiltonian on the honeycomb lattice as a minimal model for a doped spin-orbit Mott insulator. This spin-1/2 model is believed to describe the magnetic properties of the layered transition-metal oxide Na2IrO3. We determine the ground state of the system with finite charge-carrier density from the functional renormalization group (fRG) for correlated fermionic systems. To this end, we derive fRG flow equations adapted to the lack of full spin-rotational invariance in the fermionic interactions, here represented by the highly frustrated and anisotropic Kitaev exchange term. Additionally employing a set of the Ward identities for the Kitaev-Heisenberg model, the numerical solution of the flow equations suggests a rich phase diagram emerging upon doping charge carriers into the ground-state manifold (Z(2) quantum spin liquids and magnetically ordered phases). We corroborate superconducting triplet p-wave instabilities driven by ferromagnetic exchange and various singlet pairing phases. For filling delta > 1/4, the p-wave pairing gives rise to a topological state with protected Majorana edge modes. For antiferromagnetic Kitaev and ferromagnetic Heisenberg exchanges, we obtain bond-order instabilities at van Hove filling supported by nesting and density-of-states enhancement, yielding dimerization patterns of the electronic degrees of freedom on the honeycomb lattice. Further, our flow equations are applicable to a wider class of model Hamiltonians.