Magnetism and topological phases in an interacting decorated honeycomb lattice with spin-orbit coupling

We study the interplay between spin-orbit coupling (SOC) and Coulomb repulsion in a Hubbard model on a decorated honeycomb lattice which leads to a plethora of phases. While a quantum spin Hall insulator is stable at weak Coulomb repulsion and moderate SOC, a semimetallic phase emerges at large SOC in a broad range of Coulomb repulsion. This semimetallic phase has topological properties such as a finite, nonquantized spin Hall conductivity not observed in conventional metals. At large Coulomb repulsion and negligible spin-orbit coupling, electronic correlations stabilize a resonance valence bond (RVB) spin liquid state in contrast to the classical antiferromagnetic state predicted by mean-field theory. Under sufficiently strong SOC, such RVB state is transformed into a magnetic insulator consisting on S less than or similar to 3/2 localized moments on a honeycomb lattice with antiferromagnetic order and topological features.