Graph convolution with topology refinement for Automatic Reinforcement Learning

Reinforcement learning faces the challenge of sparse rewards. Existing research utilizes reward shaping based on graph convolutional neural networks (GCNs) to address this challenge. However, the automatic construction of optimal graph has been a long standing issue. Here we propose Graph Convolution with Topology Refinement for Automatic Reinforcement Learning (GTR), based on the construction of new latent graph to replace the original input graph for more effective reward shaping. It is found from this work that, the most suitable state node can be extracted through the graph entropy. Subsequently we map the original graph to subset of nodes adaptively to form a new and more compact latent graph. Since GTR utilizes trainable projection vectors for projecting all node features into one-dimensional representation, the inter-connections between the nodes of the newly constructed latent graph are consistent with the original ones. The proposed GTR stems from mathematical grounds, and preliminary experiments have shown that the proposed GTR has considerable improvement on Atari benchmark and Mujoco benchmark. Further experiment and ablation analysis have given further supports to this work.

Automated summary

This study proposes a method called Graph Convolution with Topology Refinement (GTR) for automatic reinforcement learning, which constructs a new latent graph to enhance reward shaping. The most suitable state node is identified using graph entropy, and the original graph is adaptively mapped to a subset of nodes to form a more compact latent graph. GTR utilizes trainable projection vectors for node feature projection, ensuring consistent inter-connections between nodes in the new latent graph.