Capturing the Varieties of Natural Language Inference: A Systematic Survey of Existing Datasets and Two Novel Benchmarks

Transformer-based Pre-Trained Language Models currently dominate the field of Natural Language Inference (NLI). We first survey existing NLI datasets, and we systematize them according to the different kinds of logical inferences that are being distinguished. This shows two gaps in the current dataset landscape, which we propose to address with one dataset that has been developed in argumentative writing research as well as a new one building on syllogistic logic. Throughout, we also explore the promises of ChatGPT. Our results show that our new datasets do pose a challenge to existing methods and models, including ChatGPT, and that tackling this challenge via fine-tuning yields only partly satisfactory results.

Automated summary

This article highlights the dominance of Transformer-based Pre-Trained Language Models in the field of Natural Language Inference (NLI) and identifies two gaps in the current dataset landscape. To address these gaps, the authors propose the introduction of a dataset developed in argumentative writing research and a new dataset based on syllogistic logic. The promises of ChatGPT are also explored. The results show that the new datasets pose a challenge to existing methods and models, including ChatGPT, and that fine-tuning only yields partly satisfactory results.