Optimal deep neural networks by maximization of the approximation power

We propose an optimal architecture for deep neural networks of given size. The optimal architecture obtains from maximizing the lower bound of the maximum number of linear regions approximated by a deep neural network with a ReLu activation function. The accuracy of the approximation function relies on the neural network structure characterized by the number, dependence and hierarchy between the nodes within and across layers. We show how the accuracy of the approximation improves as we optimally choose the width and depth of the network. A Monte-Carlo simulation exercise illustrates the outperformance of the optimized architecture against cross-validation methods and gridsearch for linear and nonlinear prediction models. The application of this methodology to the Boston Housing dataset confirms empirically the outperformance of our method against state-of the-art machine learning models.

Automated summary

We propose an optimal architecture for deep neural networks of given size, which optimizes the approximation of linear regions by maximizing the lower bound. The accuracy of the approximation is influenced by the structure of the neural network, including the number, dependence, and hierarchy of nodes within and across layers. Experimental results show that our optimized architecture outperforms cross-validation methods and gridsearch for linear and nonlinear prediction models, as demonstrated on the Boston Housing dataset.