Activation Function Effects and Simplified Implementation for Hopfield Neural Network

The activation function is crucial in the Hopfield neural network (HNN) to restrict the input-output relation of each neuron. The physical realizability and simplicity of the hardware circuit of activation function are beneficial to promote the practical engineering application of the HNN. However, the HNN commonly used hyperbolic tangent activation function involves a complex hardware circuit implementation. This paper discusses a piecewise-linear activation function (PWL-AF) with simplified circuit implementation and a tri-neuron small-world HNN is built as a paradigm. The hardware implementation circuit of the HNN is greatly simplified, benefited from the PWL-AF with a simple analog circuit. Meanwhile, the dynamics related to the PWL-AF and initial conditions are numerically explored. The numerical results demonstrate that the PWL-AF-based HNN can produce dynamical behaviors like the HNN based on the hyperbolic tangent activation function. Nevertheless, the multistability with up to six kinds of coexisting multiple attractors emerged because of the PWL-AF breakpoint. This can give more flexible and potential aspects in multistability-based engineering applications. Especially, the PWL-AF breakpoint value simultaneously acts as the offset booster and amplitude controller in regulating the offset boosting and amplitude rescaling of neuron states. Afterwards, an analog circuit with three straightforward operational amplifiers (op-amp)-based circuit modules is designed for the PWL-AF, and a PCB-based analog circuit is thereby implemented for the tri-neuron small-world HNN. The hardware experiments agree with the numerical simulations, implying the feasibility of the PWL-AF simplification for the HNN.

Automated summary

This paper discusses a piecewise-linear activation function (PWL-AF) with simplified circuit implementation and a tri-neuron small-world HNN is built as a paradigm. The numerical results demonstrate that the PWL-AF-based HNN can produce dynamical behaviors like the HNN based on the hyperbolic tangent activation function. The multistability with up to six kinds of coexisting multiple attractors emerged because of the PWL-AF breakpoint, which can give more flexible and potential aspects in multistability-based engineering applications.