Z2 spin liquid and chiral antiferromagnetic phase in the Hubbard model on a honeycomb lattice

In a Schwinger-fermion representation we classify all 128 possible spin liquids that preserves SU(2) spin-rotational symmetry, honeycomb lattice group symmetry, and time-reversal symmetry. Among them we identify a Z(2) spin liquid called the sublattice-pairing state (SPS) as the spin liquid phase discovered in recent numerical study on a honeycomb lattice [Meng et al., Nature (London) 464, 847 (2010)]. Our method provides a systematic way to identify spin liquids close to Mott transition. We also show that the SPS is identical to the zero-flux Z(2) spin liquid in Schwinger-boson representation [Wang, Phys. Rev. B 82, 024419 (2010)]. through an explicit duality transformation. SPS is connected to an unusual antiferromagnetic ordered phase, which we term the chiral-antiferromagnetic (CAF) phase, by an O(4) critical point. The CAF phase breaks the SU(2) spin rotational symmetry completely and has three Goldstone modes. Our results indicate that there is likely a hidden phase transition between the CAF phase and the simple antiferromagnetic phase at large U/t. We also propose numerical measurements to reveal the CAF phase and the hidden phase transition.