A novel reduced parameter s-model of estimator learning automata in the switching non-stationary environment

Learning automata (LA), a powerful tool for reinforcement learning in the field of machine learning, could explore its optimal state by continuously interacting with an external environment. Generally, the traditional LA algorithms, especially estimator LA algorithms, can be ultimately abstracted out as P- or Q-models, which are simply located in the stationary environments. A more comprehensive consideration would be S-model operating in the non-stationary environment. For this specific LA, presently the most popular achievement belongs to stochastic estimator LA (SELA). However, synchronously handing four parameters involved in SELA is an intractable job, as these parameters may vary dramatically in values under different environments, making it essential to develop a strategy for parameter tuning. In this paper, we first propose a scheme to determine the parameter searching scope and subsequently present a series of parameter searching methods, including a four-dimensional method and a two-dimensional method, making SELA applicable for any environment with switching non-stationary characteristics. Furthermore, to decrease the tuning cost, a reduced parameter SELA supported by the new two-dimensional parameter searching method emerges. And to break the traditional limit that the environmental reward probability must be symmetrically distributed, the S-model is constructed from a new perspective, thus forming a novel reduced parameter S-model of SELA (rpS-SELA). A detailed mathematical proof theoretically reveals the absolute expediency of rpS-SELA. In addition, it is demonstrated by experimental simulations that rpS-SELA outperforms others with a reduced tuning cost, a minor time consumption, a higher accuracy rate, and above all, a stronger tracking ability to the environmental switches.

Automated summary

This paper introduces a scheme to determine the parameter searching scope for SELA and subsequently presents a series of parameter searching methods, making SELA applicable for any environment with switching non-stationary characteristics. Furthermore, a reduced parameter SELA supported by a new two-dimensional parameter searching method emerges to decrease tuning cost. Experimental simulations demonstrate that rpS-SELA outperforms others with a reduced tuning cost, minor time consumption, higher accuracy rate, and a stronger tracking ability to the environmental switches.