Partial local entropy and anisotropy in deep weight spaces

We refine a recently proposed class of local entropic loss functions by restricting the smoothening regularization to only a subset of weights. The new loss functions are referred to as partial local entropies. They can adapt to the weight-space anisotropy, thus outperforming their isotropic counterparts. We support the theoretical analysis with experiments on image classification tasks performed with multilayer, fully connected, and convolutional neural networks. The present study suggests how to better exploit the anisotropic nature of deep landscapes, and it provides direct probes of the shape of the minima encountered by stochastic gradient descent algorithms. As a byproduct, we observe an asymptotic dynamical regime at late training times where the temperature of all the layers obeys a common cooling behavior.

Automated summary

In this study, a class of local entropic loss functions was refined by limiting smoothening regularization to a subset of weights, resulting in partial local entropies that can adapt to weight-space anisotropy. Experimental results supported the superiority of these new loss functions in image classification tasks compared to their isotropic counterparts. Additionally, an asymptotic dynamical regime was observed during late training times where the temperature of all layers followed a common cooling behavior.