期刊
MACHINE LEARNING
卷 111, 期 2, 页码 685-711出版社
SPRINGER
DOI: 10.1007/s10994-021-06049-9
关键词
Adversarial learning; Neural network pruning; Robustness; Sparsity
资金
- NSF China [61902245, 62032020, 62136006, 61901261]
- Science and Technology Innovation Program of Shanghai [19YF1424500, 19YF1424200]
This study demonstrates the correlation between the sparsity of network weights and model robustness, showing that sparsity improves robustness. The proposed inverse weights inheritance method enhances the robustness of large networks by inheriting weights from smaller networks and imposing sparse weights distribution.
Network pruning has been known to produce compact models without much accuracy degradation. However, how the pruning process affects a network's robustness and the working mechanism behind remain unresolved. In this work, we theoretically prove that the sparsity of network weights is closely associated with model robustness. Through experiments on a variety of adversarial pruning methods, image-classification models and datasets, we find that weights sparsity will not hurt but improve robustness, where both weights inheritance from the lottery ticket and adversarial training improve model robustness in network pruning. Based on these findings, we propose a novel adversarial training method called inverse weights inheritance, which imposes sparse weights distribution on a large network by inheriting weights from a small network, thereby improving the robustness of the large network.
作者
我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。
推荐
暂无数据