Evaluate the reliability of AI detection software

Abstract

This study examined how well AI detection software can distinguish between human-generated and AI-generated text using the Vistral-7B-Chat model. The evaluation included 30 detection tools evaluated on 10 text samples, split evenly between human and AI sources. Using descriptive statistics and ROC curve analysis, the study measured the accuracy of these tools. The results showed that the software successfully distinguished between AI and human-generated text, achieving an AUC score of 1, indicating near-perfect accuracy. The study noted variability in performance across tools, highlighting the need for continuous improvement to address interpretation and evasion challenges. This research advances our understanding of AI text detection, highlighting the urgency of robust tools to protect the integrity of human-generated content as AI technologies evolve.