Journal
ENTROPY
Volume 23, Issue 10, Pages -Publisher
MDPI
DOI: 10.3390/e23101360
Keywords
information bottleneck theory; Cascade Learning; neural networks
Categories
Funding
- EPSRC [EP/S000356/1]
- EPSRC [EP/S000356/1] Funding Source: UKRI
Ask authors/readers for more resources
Deep neural networks excel in solving challenging pattern recognition problems by forming powerful mappings between inputs and targets, and learning representations through the dynamics of learning on an information plane. Cascade Learning (CL), a method to train deep neural networks layer-by-layer, shows comparable results while saving computation and memory costs. Performance is not linked to information compression; instead, CL can inherit information about targets and gradually specialize extracted features layer-by-layer, making it a useful heuristic in setting the depth of a neural network for accurate classification.
In solving challenging pattern recognition problems, deep neural networks have shown excellent performance by forming powerful mappings between inputs and targets, learning representations (features) and making subsequent predictions. A recent tool to help understand how representations are formed is based on observing the dynamics of learning on an information plane using mutual information, linking the input to the representation (I(X;T)) and the representation to the target (I(T;Y)). In this paper, we use an information theoretical approach to understand how Cascade Learning (CL), a method to train deep neural networks layer-by-layer, learns representations, as CL has shown comparable results while saving computation and memory costs. We observe that performance is not linked to information-compression, which differs from observation on End-to-End (E2E) learning. Additionally, CL can inherit information about targets, and gradually specialise extracted features layer-by-layer. We evaluate this effect by proposing an information transition ratio, I(T;Y)/I(X;T), and show that it can serve as a useful heuristic in setting the depth of a neural network that achieves satisfactory accuracy of classification.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available