Journal
ARTIFICIAL INTELLIGENCE REVIEW
Volume 54, Issue 8, Pages 6391-6438Publisher
SPRINGER
DOI: 10.1007/s10462-021-09975-1
Keywords
Review; Neural network generalization; Overfitting; Regularization; Model simplification; Model selection; Reducing hyper-parameters; Pruning; Network compression
Categories
Ask authors/readers for more resources
This paper discusses the differences between shallow and deep neural networks in processing features, as well as the issue of overfitting. It provides a systematic review of overfitting control methods, categorizing them into passive, active, and semi-active subsets. Additionally, it highlights the adjustment of model complexity to data complexity, and the relationship between overfitting control, regularization, network compression, and network simplification.
Shallow neural networks process the features directly, while deep networks extract features automatically along with the training. Both models suffer from overfitting or poor generalization in many cases. Deep networks include more hyper-parameters than shallow ones that increase the overfitting probability. This paper states a systematic review of the overfit controlling methods and categorizes them into passive, active, and semi-active subsets. A passive method designs a neural network before training, while an active method adapts a neural network along with the training process. A semi-active method redesigns a neural network when the training performance is poor. This review includes the theoretical and experimental backgrounds of these methods, their strengths and weaknesses, and the emerging techniques for overfitting detection. The adaptation of model complexity to the data complexity is another point in this review. The relation between overfitting control, regularization, network compression, and network simplification is also stated. The paper ends with some concluding lessons from the literature.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available