Helical structures in layered magnetic superconductors due to indirect exchange interactions mediated by interlayer tunneling

Motivated by the recent discovery of helical magnetic structure in RbEuFe4As4, we investigate interlayer ordering of magnetic moments in materials composed of spatially separated superconducting and ferromagnetically aligned layers. We consider the interplay between the normal and superconducting indirect exchange interactions mediated by tunneling between the conducting layers. We elaborate a recipe to evaluate the normal interlayer interaction via two-dimensional density of states of an isolated layer and demonstrate that for bands with small fillings, such interaction is typically ferromagnetic and short range. The nearest-layer interaction is proportional to the ratio of the interlayer hopping and in-plane bandwidth squared. On the other hand, the superconducting contribution always gives antiferromagnetic interaction and may extend over several layers when the interlayer hopping energy exceeds the superconducting gap. The frustration caused by the interplay between the normal and superconducting parts may lead to spiral ground-state magnetic configuration. The fourfold in-plane anisotropy may lock the rotation angle between the moments in the neighboring layers to 90 degrees, as it was observed in RbEuFe4As4.