Toward human-level concept learning: Pattern benchmarking for AI algorithms

Artificial intelligence (AI) today is very successful at standard pattern-recognition tasks due to the availability of large amounts of data and advances in statistical data-driven machine learning. However, there is still a large gap between AI pattern recognition and human-level concept learning. Humans can learn amazingly well even under uncertainty from just a few examples and are capable of generalizing these concepts to solve new conceptual problems. The growing interest in explainable machine intelligence requires experimental environments and diagnostic/benchmark datasets to analyze existing approaches and drive progress in pattern analysis and machine intelligence. In this paper, we provide an overview of current AI solutions for benchmarking concept learning, reasoning, and generalization; discuss the state-of-the-art of existing diagnostic/benchmark datasets (such as CLEVR, CLEVRER, CLOSURE, CURI, Bongard-LOGO, V-PROM, RAVEN, Kandinsky Patterns, CLEVR-Humans, CLEVRER-Humans, and their extension containing human language); and provide an outlook of some future research directions in this exciting research domain.

Automated summary

Artificial intelligence has made significant progress in standard pattern recognition tasks, but there is still a significant gap between AI and human-level concept learning. To analyze current approaches and drive progress, experimental environments and diagnostic/benchmark datasets are needed for explainable machine intelligence. This paper provides an overview of current AI solutions for benchmarking concept learning, reasoning, and generalization, discusses state-of-the-art diagnostic/benchmark datasets, and explores future research directions in this exciting field.