Novel Spin-Liquid States in the Frustrated Heisenberg Antiferromagnet on the Honeycomb Lattice

Recent experiment on a honeycomb-lattice Heisenberg antiferromagnet (AF) Bi3Mn4O12(NO3) revealed a novel spin-liquid-like behavior down to low temperature, which was ascribed to the frustration effect due to the competition between the AF nearest-and next-nearest-neighbor interactions J(1) and J(2). Motivated by the experiment, we study the ordering of the J(1)-J(2) frustrated classical Heisenberg AF on a honeycomb lattice both by a low-temperature expansion and a Monte Carlo simulation. The model has been known to possess a massive degeneracy of the ground state, which, however, might be lifted due to thermal fluctuations leading to a unique ordered state, the effect known as order-by-disorder. We find that the model exhibits an intriguing ordering behavior, particularly near the AF phase boundary. The energy scale of the order-by-disorder is suppressed there down to extremely low temperatures, giving rise to exotic spin-liquid states like a ring-liquid or a pancake-liquid state accompanied by the characteristic spin structure factor and the field-induced antiferromagnetism. We argue that the recent experimental data are explicable if the system is in such exotic spin-liquid states.