Accurately detecting AI text when ChatGPT is told to write like a chemist

Large language models like ChatGPT can generate authentic seeming text at lightning speed, but many journal publishers reject language models as authors on manuscripts. Thus, a means to accurately distinguish human-generated from artificial intelligence (AI)-generated text is immediately needed. We recently developed an accurate AI text detector for scientific journals and, herein, test its ability in a variety of challenging situations, including on human text from a wide variety of chemistry journals, on AI text from the most advanced publicly available language model (GPT-4), and, most important, on AI text generated using prompts designed to obfuscate AI use. In all cases, AI and human text was assigned with high accuracy. ChatGPT-generated text can be readily detected in chemistry journals; this advance is a fundamental prerequisite for understanding how automated text generation will impact scientific publishing from now into the future.

Automated summary

This article introduces an accurate AI text detector that can distinguish between human-generated and AI-generated text. It has been tested in various challenging situations, including human text, AI text generated by the most advanced language model, and AI text generated using prompts to obfuscate AI use. In all cases, the AI and human text were accurately identified. This is of great importance for understanding the impact of automated text generation on scientific publishing.