Lightweight deep neural network from scratch

In general, deep neural networks (DNNs) are seriously overparameterized with enormous hardware resources demanded, which creates a heavy burden for inference applications especially for resource-constrained edge devices. To overcome this difficulty, there are two principal solutions: optimizing the overparameterized DNNs and designing high-efficiency DNN algorithms with lightweight architectures. In terms of the optimization methods, pruning is the most effective technique because the solution fundamentally optimizes the bloated DNNs by removing redundant structures from the network and can be seamlessly incorporated into all other optimization solutions as well as all kinds of DNN architectures. Nevertheless, the study reported in this paper reveals that the various excellent but also complicated pruning algorithms may not be as effective as proposals demonstrate and do not yield optimal solutions for all cases. In addition, the current lightweight DNN architectures are also overparameterized to a large extent. In this research, we propose a mechanism for determining lightweight DNN networks From Scratch (FS-DNN). First, we conduct a thorough study on the theoretical basis of evaluating the hardware resources demanded by DNNs, and establish the objective function for determining a lightweight DNN network. Based on the study, the theoretical FS-DNN for determining lightweight DNNs from scratch with high efficiency is proposed. Then, we perform a series of experiments with FS-DNN based lightweight DNNs on the public dataset CIFAR10/100 and a private dataset Kuzushiji, which prove the feasibility and efficiency of FS-DNN. According to the research, instead of adopting bloated DNN networks that demand complicated pruning algorithms to optimize the networks after the fact or the current so-called lightweight DNNs, the experimental results demonstrate that lightweight networks based on FS-DNN achieve superior performance in computing consumption with competitive or even better accuracy.

Automated summary

Deep neural networks (DNNs) are overparameterized and demanding of hardware resources, posing challenges for inference applications on resource-constrained edge devices. This study proposes a mechanism called FS-DNN for determining lightweight DNN networks from scratch, which achieves superior performance in computing consumption with competitive or even better accuracy.