Double Sparse Deep Reinforcement Learning via Multilayer Sparse Coding and Nonconvex Regularized Pruning

Deep reinforcement learning (DRL), which highly depends on the data representation, has shown its potential in many practical decision-making problems. However, the process of acquiring representations in DRL is easily affected by interference from models, and moreover leaves unnecessary parameters, leading to control performance reduction. In this article, we propose a double sparse DRL via multilayer sparse coding and nonconvex regularized pruning. To alleviate interference in DRL, we propose a multilayer sparse-coding-structural network to obtain deep sparse representation for control in reinforcement learning. Furthermore, we employ a nonconvex log regularizer to promote strong sparsity, efficiently removing the unnecessary weights with a regularizer-based pruning scheme. Hence, a double sparse DRL algorithm is developed, which can not only learn deep sparse representation to reduce the interference but also remove redundant weights while keeping the robust performance. The experimental results in five benchmark environments of the deep q network (DQN) architecture demonstrate that the proposed method with deep sparse representations from the multilayer sparse-coding structure can outperform existing sparse-coding-based DRL in control, for example, completing Mountain Car with 140.81 steps, achieving near 10% reward increase from the single-layer sparse-coding DRL algorithm, and obtaining 286.08 scores in Catcher, which are over two times the rewards of the other algorithms. Moreover, the proposed algorithm can reduce over 80% parameters while keeping performance improvements from deep sparse representations.

Automated summary

This article proposes a double sparse DRL algorithm using multilayer sparse coding and nonconvex regularized pruning to obtain deep sparse representations for control in reinforcement learning. Experimental results demonstrate its superior performance in control compared to existing sparse-coding-based DRL methods, with significant rewards increase and parameter reduction.