Deep Reinforcement Learning for Optimal Hydropower Reservoir Operation

Optimal operation of hydropower reservoir systems is a classical optimization problem of high dimensionality and stochastic nature. A key challenge lies in improving the interpretability of operation strategies, i.e., the cause-effect relationship between system outputs (or actions) and contributing variables such as states and inputs. This paper reports for the first time a new deep reinforcement learning (DRL) framework for optimal operation of reservoir systems based on deep Q-networks (DQNs), which provides a significant advance in understanding the performance of optimal operations. DQN combines Q-learning and two deep artificial neural networks (ANNs), and acts as the agent to interact with the reservoir system through learning its states and providing actions. Three knowledge forms of learning considering the states, actions, and rewards were constructed to improve the interpretability of operation strategies. The impacts of these knowledge forms and DRL learning parameters on operation performance were analyzed. The DRL framework was tested on the Huanren hydropower system in China, using 400-year synthetic flow data for training and 30-year observed flow data for verification. The discretization levels of reservoir water level and energy output yield contrasting effects: finer discretization of water level improved performance in terms of annual hydropower generated and hydropower production reliability; however, finer discretization of hydropower production can reduce search efficiency, and thus the resulting DRL performance. Compared with benchmark algorithms including dynamic programming, stochastic dynamic programming, and decision tree, the proposed DRL approach can effectively factor in future inflow uncertainties when determining optimal operations and can generate markedly higher hydropower. This study provides new knowledge of the performance of DRL in the context of hydropower system characteristics and data input features, and shows promise for potentially being implemented in practice to derive operation policies that can be updated automatically by learning from new data.

Automated summary

This study introduces a new deep reinforcement learning (DRL) framework for optimal operation of hydropower reservoir systems, which combines Q-learning and two deep artificial neural networks for improved interpretability and performance. Through analysis of knowledge forms and DRL learning parameters, as well as testing on the Huanren hydropower system, the study reveals the impacts of discretization levels on operation performance and the advantages of the DRL approach over benchmark algorithms.