Frustration and entanglement in the t2g spin-orbital model on a triangular lattice:: Valence-bond and generalized liquid states

We consider the spin-orbital model for a magnetic system with singly occupied but triply degenerate t(2g) orbitals coupled into a planar, triangular lattice, as would be exemplified by NaTiO2. We investigate the ground states of the model for interactions which interpolate between the limits of pure superexchange and purely direct-exchange interactions. By considering ordered and dimerized states at the mean-field level, and by interpreting the results from exact diagonalization calculations on selected finite systems, we demonstrate that orbital interactions are always frustrated and that orbital correlations are dictated by the spin state, manifesting an intrinsic entanglement of these degrees of freedom. In the absence of Hund coupling, the ground state changes from a highly resonating, dimer-based, symmetry-restored spin and orbital-liquid phase to one based on completely static, spin-singlet valence bonds. The generic properties of frustration and entanglement survive even when spins and orbitals are nominally decoupled in the ferromagnetic phases stabilized by a strong Hund coupling. By considering the same model on other lattices, we discuss the extent to which frustration is attributable separately to geometry and to interaction effects.