Spin susceptibility and helical magnetic order at the edges/surfaces of topological insulators due to Fermi surface nesting

We study spin susceptibility andmagnetic order at the edges/surfaces of two-and three-dimensional topological insulators when the Fermi surface is nested. We find that, due to spin-momentum locking as well as time-reversal symmetry, spin susceptibility at the nesting wave vector has a strong helical feature. It follows then that a helical spin density wave state emerges at low temperature due to Fermi surface nesting. The helical feature of spin susceptibility also has a profound impact on the magnetic order in magnetically doped surface of three-dimensional topological insulators. In such a system, from the mean-field Zener theory, we predict a helical magnetic ordered state.