Kane-Mele Hubbard model on a zigzag ribbon: Stability of the topological edge states and quantum phase transitions

We study the quantum phases and phase transitions of the Kane-Mele Hubbard (KMH) model on a zigzag ribbon of honeycomb lattice at a finite size via the weak-coupling renormalization group (RG) approach. In the noninteracting limit, the Kane-Mele (KM) model is known to support topological edge states where electrons show helical property with orientations of the spin and momentum being locked. The effective interedge hopping terms are generated due to finite-size effect. In the presence of an on-site Coulomb (Hubbard) interaction and the interedge hoppings, special focus is put on the stability of the topological edge states (TI phase) in the KMH model against (i) the charge and spin gaped (II) phase, (ii) the charge gaped but spin gapless (IC) phase, and (iii) the spin gaped but charge gapless (CI) phase depending on the number (even/odd) of the zigzag ribbons, doping level (electron filling factor) and the ratio of the Coulomb interaction to the interedge tunneling. We discuss different phase diagrams for even and odd numbers of zigzag ribbons. We find the TI-CI, II-IC, and II-CI quantum phase transitions are of the Kosterlitz-Thouless (KT) type. By computing various correlation functions, we further analyze the nature and leading instabilities of these phases. The relevance of our results for graphene is discussed.