Unsupervised neural networks for the identification of minimum overcomplete basis in visual data

We define a minimum overcomplete basis as that set of basis vectors which has more members than necessary to span the space but which minimises an energy function of the space. We present an unsupervised artificial neural network that may be used to identify a minimum overcomplete basis for a data set. Building on a network which normally finds the principal components of data we have previously shown that, by applying a non-linearity which half-wave rectifies the outputs of this network, a much sparser response is achieved. The restriction of the coding to positive values necessitates an overcomplete representation. By adding Gaussian noise to the outputs after this rectification we may control the dimensionality of the overcomplete basis so that a minimal basis set is formed under the positive coding constraint. The operation of the network is demonstrated by the application of the network to artificial and natural image data. (C) 2002 Elsevier Science B.V. All rights reserved.