Theory of quasiparticle scattering in a two-dimensional system of helical Dirac fermions: Surface band structure of a three-dimensional topological insulator

We study the quasiparticle interference (QPI) patterns caused by scattering off nonmagnetic, magnetic point impurities, and edge impurities, separately, in a two-dimensional helical liquid, which describes the surface states of a topological insulator. The unique features associated with hexagonal warping effects are identified in the QPI patterns of charge density with nonmagnetic impurities and spin density with magnetic impurities. The symmetry properties of the QPI patterns can be used to determine the symmetry of microscopic models. The Friedel oscillation is calculated for edge impurities and the decay of the oscillation is not universal, strongly depending on Fermi energy. Some discrepancies between our theoretical results and current experimental observations are discussed.