Proximal Policy Optimization for Energy Management of Electric Vehicles and PV Storage Units

Connected autonomous electric vehicles (CAEVs) are essential actors in the decarbonization process of the transport sector and a key aspect of home energy management systems (HEMSs) along with PV units, CAEVs and battery energy storage systems. However, there are associated uncertainties which present new challenges to HEMSs, such as aleatory EV arrival and departure times, unknown EV battery states of charge at the connection time, and stochastic PV production due to weather and passing cloud conditions. The proposed HEMS is based on proximal policy optimization (PPO), which is a deep reinforcement learning algorithm suitable for continuous complex environments. The optimal solution for HEMS is a tradeoff between CAEV driver's range anxiety, batteries degradation, and energy consumption, which is solved by means of incentives/penalties in the reinforcement learning formulation. The proposed PPO algorithm was compared to conventional methods such as business-as-usual (BAU) and value iteration (VI) solutions based on dynamic programming. Simulation results indicate that the proposed PPO's performance showed a daily energy cost reduction of 54% and 27% compared to BAU and VI, respectively. Finally, the developed PPO algorithm is suitable for real-time operations due to its fast execution and good convergence to the optimal solution.

Automated summary

Connected autonomous electric vehicles (CAEVs) play a crucial role in the decarbonization of the transport sector and are integral to home energy management systems (HEMSs) alongside PV units and battery energy storage systems. However, uncertainties associated with CAEVs pose challenges to HEMSs, such as uncertain arrival and departure times, unknown battery states of charge, and variable PV production. A proposed HEMS based on proximal policy optimization (PPO) addresses these challenges through deep reinforcement learning. Simulation results demonstrate that the PPO algorithm outperforms conventional methods, achieving a significant daily energy cost reduction.