Deep Learning Architecture Improvement Based on Dynamic Pruning and Layer Fusion

The heavy workload of current deep learning architectures significantly impedes the application of deep learning, especially on resource-constrained devices. Pruning has provided a promising solution to compressing the bloated deep learning models by removing the redundancies of the networks. However, existing pruning methods mainly focus on compressing the superfluous channels without considering layer-level redundancies, which results in the channel-pruned models still suffering from serious redundancies. To mitigate this problem, we propose an effective compression algorithm for deep learning models that uses both the channel-level and layer-level compression techniques to optimize the enormous deep learning models. In detail, the channels are dynamically pruned first, and then the model is further optimized by fusing the redundant layers. Only a minor performance loss results. The experimental results show that the computations of ResNet-110 are reduced by 80.05%, yet the accuracy is only decreased by 0.72%. Forty-eight convolutional layers could be discarded from ResNet-110 with no loss of performance, which fully demonstrates the efficiency of the proposal.

Automated summary

Existing compression methods for deep learning models mainly focus on reducing redundant channels and ignore layer-level redundancies, resulting in compressed models that still have significant redundancies. To address this issue, we propose an effective compression algorithm that combines channel-level and layer-level compression techniques to optimize deep learning models. Experimental results show that the proposed algorithm reduces computations by 80.05% with only a 0.72% decrease in accuracy. This demonstrates the efficiency of the algorithm, as 48 convolutional layers can be discarded with no performance loss.