A Paradigm Shift from Human Writing to Machine Generation in Personality Test Development: an Application of State-of-the-Art Natural Language Processing

Natural language processing (NLP) techniques have become increasingly popular in areas of psychological assessment. Recently, researchers have sought to use NLP techniques for automatic item generation (AIG) in the personality domain. Nevertheless, NLP-based approaches to personality AIG are new and many questions are still unanswered. Our research builds upon previous illustrations of AIG in personality in several ways. First, we applied a prompt-based generative pre-trained transformer 3 (GPT-3) to generate personality items. This approach provides several practical advantages for researchers and practitioners compared to previous AIG approaches. Second, we thoroughly compared various psychometric properties between machine- and human-authored personality items. Lastly, we examined the measurement invariance of machine-authored personality items between gender groups to ensure fair organizational decision-making. Results revealed that the machine-authored personality items provided good psychometric properties and little measurement biases between genders. Practical considerations, contributions, and future research directions of the AIG technique for non-cognitive tests were discussed.

Automated summary

This research explores the application of natural language processing techniques in automatic item generation for personality assessment. It utilizes a prompt-based generative pre-trained transformer to generate personality items and compares their psychometric properties with human-authored items. The study also examines the measurement invariance of machine-authored items between gender groups.