Actor-critic learning for optimal building energy management with phase change materials

Energy management in buildings using phase change materials (PCM) to improve thermal performance is challenging due to the nonlinear thermal capacity of the PCM. To address this problem, this paper adopts a model-free actor-critic on-policy reinforcement learning method based on deep deterministic policy gradient (DDPG). The proposed approach overcomes the major weakness of model-based approaches, such as approximate dynamic programming (ADP), which require an explicit thermal model of the building under control. This requirement makes a plug-and-play implementation of the energy management algorithm in an existing smart meter difficult due to the wide variety of building design and construction types. To overcome this difficulty, we use a DDPG algorithm that can learn policies in continuous action spaces without access to the full dynamics of the building. We demonstrate the competitive performance of DDPG by benchmarking it against an ADP-based approach with access to the full thermal dynamics of the building.