Reinforcement learning for electricity dispatch in grids with high intermittent generation and energy storage systems: A case study for the Brazilian grid

Intermittent energy sources such as wind and solar have recently been growing a lot faster than dispatchable energy sources in Brazil, which made investments in energy storage systems become an attractive possibility in the country. Current operational policies for energy dispatch do not consider storage systems and need adjustments to fit this technology. With this motivation, we use reinforcement learning techniques to develop policies for managing storage systems in a grid that can handle time-varying inputs and loads, with rolling forecasts. We use a deterministic lookahead (DLA) policy which has been parametrically modified to perform well in the presence of uncertain forecasts. For realistic simulations, the base model considers important characteristics in a grid that influence the interaction between scheduling and real-time operation such as power and ramping capacities, notification times, and stochastic forecasts. The parametric modification with tunable parameters allows an optimal balance between two conflicting services provided by the storage system: time-shifting and spinning reserves. Optimal reserves ranged from 35% to 100%, depending on the tested dataset, which shows the importance of tuning. Differently from stochastic lookahead policies, which are computationally expensive, parameterized DLA policies can be applied to real-time operation after being optimized in a stochastic base model.