On the Use of Low-discrepancy Sequences in the Training of Neural Networks

The quasi-Monte Carlo methods use specially designed deterministic sequences with improved uniformity properties compared with random numbers, in order to achieve higher rates of convergence. Usually certain measures like the discrepancy are used in order to quantify these uniformity properties. The usefulness of certain families of sequences with low discrepancy, like the Sobol and Halton sequences, has been established in problems with high practical value as in Mathematical Finance. Multiple studies have been done about applying these sequences also in the domains of optimisation and machine learning. Currently many types of neural networks are used extensively to achieve break-through results in Machine Learning and Artificial Intelligence. The process of training these networks requires substantial computational resources, usually provided by using powerful GPUs or specially designed hardware. In this work we study different approaches to employ efficiently low-discrepancy sequences at various places in the training process where their uniformity properties can speed-up or improve the training process. We demonstrate the advantage of using Sobol low-discrepancy sequences in benchmark problems and we discuss various practical issues that arise in order to achieve acceptable performance in real-life problems.

Automated summary

Quasi-Monte Carlo methods utilize specially designed deterministic sequences to achieve higher convergence rates compared to random numbers. The usefulness of low-discrepancy sequences, such as Sobol and Halton sequences, has been established in various fields including Mathematical Finance, optimization, and machine learning. This study explores different approaches to efficiently employ low-discrepancy sequences in the training process of neural networks, highlighting their advantage in benchmark problems and discussing practical issues in real-life applications.