Interplay of Fulde-Ferrell-Larkin-Ovchinnikov and vortex states in two-dimensional superconductors

Clean superconductors with weakly coupled conducting planes have been suggested as promising candidates for observing the Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) state. We consider here a layered superconductor in a magnetic field of arbitrary orientation with respect to the conducting plane. In this case there is competition of Pauli spin-pair-breaking effects, favoring the FFLO state, and orbital-pair-breaking effects favoring the Abrikosov vortex state. In previous work, phase transitions to phases with pairing in Landau levels with quantum numbers n > 0 have been predicted. Here, we calculate the actual structure of the stable below H-c2 by minimizing the free energy. We find new order parameter structures differing from both the traditional Abrikosov and FFLO solutions. These include two-dimensional periodic structures with several zeros of the order parameter, as well as quasi-one-dimensional structures consisting of vortex chains separated by FFLO domains. We discuss the limit of high n, where some interesting but yet unsolved questions appear.