StoryScope: Investigating idiosyncrasies in AI fiction
| Authors | Jenna Russell et al. |
| Year | 2026 |
| Field | NLP |
| arXiv | 2604.03136 |
| Download | |
| Categories | cs.CL |
Abstract
As AI-generated fiction becomes increasingly prevalent, questions of authorship and originality are becoming central to how written work is evaluated. While most existing work in this space focuses on identifying surface-level signatures of AI writing, we ask instead whether AI-generated stories can be distinguished from human ones without relying on stylistic signals, focusing on discourse-level narrative choices such as character agency and chronological discontinuity. We propose StoryScope, a pipeline that automatically induces a fine-grained, interpretable feature space of discourse-level narrative features across 10 dimensions. We apply StoryScope to a parallel corpus of 10,272 writing prompts, each written by a human author and five LLMs, yielding 61,608 stories, each ~5,000 words, and 304 extracted features per story. Narrative features alone achieve 93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution, retaining over 97% of the performance of models that include stylistic cues. A compact set of 30 core narrative features captures much of this signal: AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. Per-model fingerprint features enable six-way attribution: for example, Claude produces notably flat event escalation, GPT over-indexes on dream sequences, and Gemini defaults to external character description. We find that AI-generated stories cluster in a shared region of narrative space, while human-authored stories exhibit greater diversity. More broadly, these results suggest that differences in underlying narrative construction, not just writing style, can be used to separate human-written original works from AI-generated fiction.
Engineering Breakdown
Plain English
StoryScope is a system that automatically detects whether stories are written by humans or AI by analyzing narrative structure rather than writing style. The researchers built a pipeline that extracts interpretable features across 10 dimensions of narrative discourse—like character agency and timeline consistency—and tested it on a massive parallel corpus of 61,608 stories (about 5,000 words each) generated from 10,272 prompts written by humans and five different LLMs. The key insight is that AI-generated fiction leaves detectable fingerprints in how stories unfold narratively, not just in word choice or sentence structure. This shifts the focus from surface-level detection (which is easy to evade) to deeper structural patterns that are harder to manipulate.
Core Technical Contribution
The core contribution is the StoryScope pipeline itself—a method for automatically inducing and extracting fine-grained, interpretable narrative features at the discourse level rather than the syntactic level. Rather than relying on stylistic classifiers (which can be fooled by simple paraphrasing), the authors identified that narrative-level choices (character agency, chronological consistency, plot structure) reveal systematic differences between human and AI authorship. This is novel because prior work on AI detection focused on surface signals (word frequency, sentence length, rare n-grams), whereas StoryScope operates on a higher level of abstraction that reflects fundamental differences in how humans and LLMs construct stories. The 10-dimensional feature space is interpretable, meaning engineers can understand why a story is flagged as AI-generated, not just that it is.
How It Works
StoryScope processes each story through multiple stages: first, it ingests the raw narrative text and performs discourse parsing to identify narrative structure elements (temporal markers, character mentions, causal relationships). Second, it computes features across 10 discourse dimensions—examples likely include metrics for character consistency, agency distribution over time, narrative coherence, plot branching, emotional arc monotonicity, chronological discontinuities, and dialogue-to-narration balance. Third, it aggregates these features into vectors that capture the narrative "shape" of the story independent of lexical content. Finally, these feature vectors are compared between human and LLM-generated stories to identify systematic differences. The pipeline is applied to a parallel corpus where each of the 10,272 writing prompts has both a human version and five LLM versions (GPT, Claude, etc. presumably), creating a controlled experimental setup where prompt and domain are held constant and only the author type varies.
Production Impact
For production systems, StoryScope offers a more robust AI detection mechanism for content moderation platforms, literary publishing workflows, and academic integrity checking in creative writing. Rather than deploying a black-box classifier, you get interpretable features that let you explain to users or content creators why their submission was flagged as AI-generated—this is critical for fairness and transparency in high-stakes applications. The approach could reduce false positives that plague stylistic classifiers, since narrative structure is harder to fake than surface-level style (which LLMs can mimic well). However, there are trade-offs: the pipeline requires full story parsing and discourse analysis (computationally more expensive than bag-of-words approaches), longer texts to extract reliable features (~5,000 words minimum based on the corpus), and retraining as LLMs improve at mimicking human narrative structure. Integration requires building or integrating a robust discourse parser, which adds latency and complexity compared to simpler detection methods.
Limitations and When Not to Use This
The study is limited to relatively long-form fiction (~5,000 words per story), so it's unclear how well StoryScope generalizes to shorter stories, poetry, or non-narrative prose where discourse structure may be less pronounced. The corpus uses only five specific LLMs and a single source of human writing, so the features may not distinguish human-written fiction from future LLMs that are trained specifically to mimic human narrative patterns or from specialized fiction models. The paper likely assumes clear temporal and causal structures in stories; edge cases like experimental fiction, postmodern narratives, or stories without linear time may confound the feature extraction. Adversarial LLMs that are fine-tuned to match human narrative distributions on these 10 dimensions could defeat this detection method, making it an arms race rather than a permanent solution. The approach requires ground-truth labels of human vs. AI authorship for validation, which becomes harder to obtain as AI-generated content becomes more prevalent and indistinguishable.
Research Context
This work builds on prior literature in authorship attribution (which traditionally uses stylometric features) and AI text detection, but shifts the focus from shallow stylistic signals to deeper semantic and structural properties. It contributes to the growing concern around AI-generated content authentication as LLMs become better at imitating human writing—prior approaches focused on detectable artifacts like repeated phrases or unusual n-gram distributions, but StoryScope shows those can be gamed. The paper opens a research direction into what aspects of narrative are fundamentally human vs. mechanistically generated, potentially informing both better detection methods and better LLM training for fiction. The large parallel corpus (61,608 stories) is also a valuable resource for the community to study differences between human and AI fiction, likely enabling follow-up work on style transfer, narrative generation, and understanding narrative biases in LLMs.
:::tip Subscribe Get weekly breakdowns of papers like this in AI Letters - the newsletter for engineers building production AI systems. :::