Next-generation deep learning based on simulators and synthetic data

Deep learning (DL) is being successfully applied across multiple domains, yet these models learn in a most artificial way: they require large quantities of labeled data to grasp even simple concepts. Thus, the main bottleneck is often access to supervised data. Here, we highlight a trend in a potential solution to this challenge: synthetic data. Synthetic data are becoming accessible due to progress in rendering pipelines, generative adversarial models, and fusion models. Moreover, advancements in domain adaptation techniques help close the statistical gap between synthetic and real data. Paradoxically, this artificial solution is also likely to enable more natural learning, as seen in biological systems, including continual, multimodal, and embodied learning. Complementary to this, simulators and deep neural networks (DNNs) will also have a critical role in providing insight into the cognitive and neural functioning of biological systems. We also review the strengths of, and opportunities and novel challenges associated with, synthetic data.

Automated summary

Deep learning has achieved success in various domains, but the requirement for large amounts of labeled data presents a major bottleneck. Synthetic data is emerging as a potential solution, aided by advances in rendering pipelines, generative adversarial models, and fusion models. Domain adaptation techniques are also closing the statistical gap between synthetic and real data. The use of synthetic data and deep neural networks provides insights into the cognitive and neural functioning of biological systems.