Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers

🤖 Plain-English Summary

Large language models such as ChatGPT can produce increasingly realistic text, with unknown information on the accuracy and integrity of using these models in scientific writing. Depending on publisher-specific guidelines, AI output detectors may serve as an editorial tool to help maintain scientific standards.

🔑 Key Findings

• We gathered fifth research abstracts from five high-impact factor medical journals and asked ChatGPT to generate research abstracts based on their titles and journals.
• Most generated abstracts were detected using an AI output detector, 'GPT-2 Output Detector', with % 'fake' scores (higher meaning more likely to be generated) of median [interquartile range] of 99.98% 'fake' [12.73%, 99.98%] compared with median 0.02% [IQR 0.02%, 0.09%] for the original abstracts.
• The AUROC of the AI output detector was 0.94.

🔗 Related Resources