Non-classical large deviations for a noisy system with non-isolated attractors

We study the large deviations of a simple noise-perturbed dynamical system having continuous sets of steady states, which mimic those found in some partial differential equations related, for example, to turbulence problems. The system is a two-dimensional nonlinear Langevin equation involving a dissipative, non-potential force, which has the essential effect of creating a line of stable fixed points (attracting line) touching a line of unstable fixed points (repelling line). Using different analytical and numerical techniques, we show that the stationary distribution of this system satisfies, in the low-noise limit, a large deviation principle containing two competing terms: (i) a 'classical' but subdominant large deviation term, which can be derived from the Freidlin-Wentzell theory of large deviations by studying the fluctuation paths or instantons of the system near the attracting line, and (ii) a dominant large deviation term, which does not follow from the Freidlin-Wentzell theory, as it is related to fluctuation paths of zero action, referred to as sub-instantons, emanating from the repelling line. We discuss the nature of these sub-instantons, and show how they arise from the connection between the attracting and repelling lines. We also discuss in a more general way how we expect these to arise in more general stochastic systems having connected sets of stable and unstable fixed points, and how they should determine the large deviation properties of these systems.