Detecting AI assisted submissions in introductory programming via code anomaly

Abstract

AbstractView references

Artificial Intelligence (AI) can foster education but can also be misused to breach academic integrity. Large language models like ChatGPT are able to generate solutions for individual assessments that are expected to be completed independently. There are a number of automated detectors for AI assisted work. However, most of them are not dedicated to programming and/or they rely on existing student submissions (i.e., the learning approach). This paper presents a straightforward detector for AI assisted code, relying on code anomaly. No existing student submissions are needed. The detector employs 34 features covering constants, data structures, branches, loops, functions, and others. According to our evaluation on three data sets, the detector and its normalized variation are effective with 89% top-K precision. However, allowing discussion among colleagues and access to the internet might reduce the effectiveness by 25%. The effectiveness is further reduced by about the same amount when AI assistance is only used on some tasks, not the whole submissions. Although our detectors should be used with caution due to the limitations, it sufficiently shows that code anomaly can be distinctive for identifying AI assisted work. Instructors can start looking for the code anomaly among the submissions for such identification. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.