Spiral order by disorder and lattice nematic order in a frustrated Heisenberg antiferromagnet on the honeycomb lattice

Motivated by recent experiments on Bi3Mn4O12(NO3), we study a frustrated J(1)-J(2) Heisenberg model on the two dimensional (2D) honeycomb lattice. The classical J(1)-J(2) Heisenberg model on the 2D honeycomb lattice exhibits Neel order for J(2) < J(1)/6. For J(2) > J(1)/6, it has a family of degenerate incommensurate spin spiral ground states where the spiral wave vector can point in any direction. Spin wave fluctuations at leading order lift this accidental degeneracy in favor of specific wave vectors, leading to spiral order by disorder. For spin S = 1/2, quantum fluctuations are, however, likely to be strong enough to melt the spiral order parameter over a wide range of J(2)/J(1). Over a part of this range, we argue that the resulting state is a valence bond solid (VBS) with staggered dimer order - this VBS is a lattice nematic which breaks lattice rotational symmetry. Our arguments are supported by comparing the spin wave energy with the energy of the VBS obtained using a bond operator formalism. Turning to the effect of thermal fluctuations on the spiral ordered state, any nonzero temperature destroys the magnetic order, but the discrete rotational symmetry of the lattice remains broken resulting in a thermal analogue of the nematic VBS. We present arguments, supported by classical Monte Carlo simulations, that this nematic transforms into the high temperature paramagnet via a thermal phase transition which is in the universality class of the classical 3-state Potts (clock) model in 2D. We discuss the relevance of our results for honeycomb magnets, such as Bi3M4O12(NO3) (with M=Mn,V,Cr), and bilayer triangular lattice magnets.