Readers Rate ChatGPT Stories Higher Than Human Ones Until They Learn the AI Origin

According to a study by Sydney Sears and Deena Skolnick Weisberg published in the journal Judgment and Decision Making, readers are unable to tell whether a short story was written by a human or generated by ChatGPT. In three experiments with over 2500 participants, subjects performed no better than chance at distinguishing human-written from AI-generated fictional short stories. In the first experiment, each of 1682 participants read one of six stories, three from literary magazines and three generated with ChatGPT 4.0, with prompts based on theme, style, and narrative perspective of the human originals. Half were told the story was human-written, half that it was from ChatGPT. AI-generated stories scored significantly higher on perceived quality and immersion (mean quality 1.54 vs 0.97; immersion 1.42 vs 1.00). However, participants who were told a story was human-written rated it higher, regardless of actual authorship. Those with positive attitudes toward AI gave generally higher ratings, and their ratings increased further when told the story came from ChatGPT, while AI-skeptic participants showed the opposite effect. In two further experiments with 905 participants, each read both human and AI stories and had to identify which was which, but were still no better than chance. Self-reported experience with AI systems correlated positively with correct attribution, while experience with fiction did not help. The authors suggest AI texts are typically more fluent, easier to understand, and emotionally more positive, which may explain higher ratings without being literarily better, since quality fiction is often intentionally challenging. The researchers conclude that AI can produce creative works perceived as at least equivalent to human work, but people do not credit AI for it. Data and materials are openly available on the Open Science Framework.