Time-reversal symmetry breaking at the edge states of a three-dimensional topological band insulator

We study the time-reversal (T) symmetry breaking of two-dimensional helical Fermi liquid, with application to the edge states of three-dimensional (3D) topological band insulators with only one two-component Dirac fermion at finite chemical potential, as well as other systems with spin-orbit coupling. The T-breaking Ising order parameter is not overdamped and the theory is different from the ordinary Hertz-Millis theory for order parameters at zero momentum. We argue that the T-breaking phase transition is a 3D Ising transition, and the quasiparticles are well defined in the quantum critical regime.