LEXINGTON, Va. April 5, 2024 — Dr. Donald McCabe, professor emeritus at Rutgers Business School, researched cheating among 1,800 college students, and found that 15% turned in a fake term paper, 84% cheated on written assignments, and 52% plagiarized one or more sentences for a paper. Having to check for plagiarism on every student submitted paper, only adds to the already burdensome workload for teachers. So, what is the solution? Abigail Basener ’24, a Virginia Military Institute cadet majoring in applied mathematics, addressed the topic in her honors thesis, “Detecting AI-Generated Writing vs. Student Writing.”  

Basener’s research goals were to make a model to predict if writing is human or generated by a large langue model (LLM), a computer program that learns and generates human-like language by finding patterns in the ways words go together and guess the next most likely words; to understand how the model works to gain insight into LLMs; and to better inform users of the pitfalls and strengths of LLMs and their detectors.   

Basener explained that LLMs are trained on very large datasets, often the internet. “The important thing to know is a LLM looks at how language is put together, but it doesn’t actually know anything. For example, it sees the words Delta and airlines often together, but it doesn't understand what Delta is or what an airline is.”  

The first thing Basener did in designing her model was to decide on a set of 21 features (how she talked to the model) to use. “The idea is if I had an orange, and I asked you to tell me what type of fruit it was, you wouldn’t tell me it’s an orange, you’d look at the shape, the color, the size, characteristics like that,” she explained. The features included the number of sentences per paragraph, words per paragraph, sentence length, and the number of occurrences of certain words and characters.  

Basener used 2,000 entries of data to test her model: One thousand from human writing and 1,000 from artificial intelligence (AI) writing. She discovered that human writing contains more questions marks than AI writing. AI prefers to use the word, “however,” and humans prefer to use the less formal word “but.” Humans prefer to use the word “this” more often than AI, and human writing has shorter and longer sentences than AI, which prefers to use sentences of medium length. The more sentences in a paragraph, the greater the odds that the writing is AI generated. 

Basener ran one of her own samples of writing through her model, along with the same sample she first ran through ChatGPT, an LLM that enables users to refine their writing to a desired length, format, style, level of detail, and language. Her model successfully recognized her writing as human and the ChatGPT version as AI.   

She concluded by stating, “By using model analysis, we can better understand how detectors work.” She also found that human writing tends to be more emotional and varied in structure. “An LLM does not recognize poetry, and cannot write poetry.” 

Her advisor on the project, Col. John David, professor in the Department of Applied Mathematics, stated that Basener’s project was timely and technically challenging. “Beyond being an accurate detector, the really interesting part is how it begins to show us the difference between human and AI writing. It is one of the best honors projects I’ve ever seen,” he said. 

Basener, who is minoring in economics, is the daughter of Amber and William Basener of Advance Mills, Virginia, where she was homeschooled. She holds an associate degree in computer science from Piedmont Virginia Community College. After graduation, she plans on pursuing a master’s degree in machine learning, the study of the mathematics and computer science that goes into building AI, followed by a PhD.  

Marianne Hause
Communications & Marketing
VIRGINIA MILITARY INSTITUTE