3.8 Proceedings Paper

Quasi-orthogonality and intrinsic dimensions as measures of learning and generalisation

出版社

IEEE
DOI: 10.1109/IJCNN55064.2022.9892337

关键词

deep learning; neural architectural search; orthogonality; dimensionality

资金

  1. UKRI Turing AI Acceleration Fellowship: Adaptive, Robust, and Resilient AI Systems for the FuturE [EP/V025295/1]
  2. UK Trustworthy Autonomous Systems Verifiability Node [EP/V026801/1]

向作者/读者索取更多资源

The study suggests a correlation between the accuracies of trained networks and values of easily computable measures defined on randomly initialised networks, raising the question of exploring other, more principled measures. The dimensionality and quasi-orthogonality of neural networks' feature space may jointly serve as discriminants of network performance.
Finding the best architectures for learning machines, such as deep neural networks, is a well-known technical and theoretical challenge. Recent work by Mellor et al [1] showed that there may exist correlations between the accuracies of trained networks and the values of some easily computable measures defined on randomly initialised networks which may enable the search of tens of thousands of neural architectures without training. Mellor et al [1] used the Hamming distance evaluated over all the ReLU neurons as such a measure. Motivated by these findings, in our work, we ask the question of the existence of other and perhaps more principled measures which could be used as determinants of the potential success of a given neural architecture. In particular, we examine if the dimensionality and quasi-orthogonality of neural networks' feature space could be correlated with the network's performance after training. We showed, using the setup as in Mellor et al [1], that dimensionality and quasi-orthogonality may jointly serve as networks' performance discriminants. In addition to offering new opportunities to accelerate neural architecture search, our findings suggest important relationships between the networks' final performance and properties of their randomly initialised feature spaces: data dimension and quasi-orthogonality.

作者

我是这篇论文的作者
点击您的名字以认领此论文并将其添加到您的个人资料中。

评论

主要评分

3.8
评分不足

次要评分

新颖性
-
重要性
-
科学严谨性
-
评价这篇论文

推荐

暂无数据
暂无数据