StoryScope: Investigating idiosyncrasies in AI fiction

2026-06-22 · source · source ↗ · raw capture · wiki page

Russell et al. (University of Maryland / Google DeepMind) introduce StoryScope, a pipeline that automatically extracts 304 interpretable discourse-level narrative features across 10 dimensions (plot, agents, temporal structure, etc.) from fiction. Applied to a parallel corpus of 10,272 writing prompts each written by a human and five LLMs (Claude, DeepSeek, Gemini, GPT, Kimi) — 61,608 stories total, ~5,000 words each — they show that narrative features alone achieve 93.2% macro-F1 for human vs. AI detection, retaining 97% of performance of models that also include stylistic cues. A compact set of 30 core features captures the signal: AI stories over-explain themes, favor tidy single-track plots, and render emotion through bodies rather than naming it; human stories are temporally complex, morally ambiguous, and acknowledge the reader. Per-model fingerprints emerge (Claude: flat escalation, no dreams; GPT: gossip-driven plots; Gemini: external character description). AI stories cluster in a shared narrative region; human stories are rarer and more dispersed. The features survive stylistic editing (only 1.6pp F1 drop after LAMP artifact removal), suggesting narrative structure is a more durable detection signal than surface prose patterns.

Key Findings

Detection performance

Core human-AI differences

AI over-explains themes. AI stories score ~20% higher on thematic explicitness (1-5 scale). Narrators explicitly state the theme 77% of the time vs. 52% for humans. AI dialogue serves philosophical debate (59% vs. 34%). AI references to other works are vague allusions (72% vs. 50%) rather than specific named references.

AI renders emotion through bodies. AI conveys emotion through physical sensations 81% of the time vs. 38% human. AI uses smell-based imagery more (82% vs. 57%), treats setting as psychological mirror more heavily. Human authors use explicit emotion labels 29% of the time vs. just 8% for AI.

Humans subvert linearity. Human stories have more time jumps, flashbacks, nonlinear structure. AI favors single-track narratives with tight causal chains (4.20 vs. 3.92 on causality scale), protagonist-driven resolutions (69% vs. 46%), and far fewer subplots (79% "no subplots" vs. 57%). AI resolutions favor internal understanding/acceptance (47% vs. 27%).

Humans engage the outside world. Humans reference specific texts/authors at nearly double the AI rate (47% vs. 24%). Humans break the fourth wall more often (67% vs. 39%), address readers directly more (28% vs. 7%). Humans use balanced mix of explicit/implicit references (37% vs. 16%).

Human stories are rarer and more diverse. Mean rarity percentile 0.71 human vs. 0.49 AI (Cohen's d = 0.83). 24.7% of human stories fall in the top 10% rarest stories corpus-wide, vs. 7.1% of AI stories. At the prompt level, the human story is the rarest of all six versions 57.8% of the time.

Per-model fingerprints

AI convergence

The five AI models occupy overlapping regions of narrative feature space, well-separated from human stories. Mean human-AI centroid distance is 1.6× the mean AI-AI centroid distance (6.6 vs. 4.3). Even the closest human-AI centroid pair is farther apart than the most distant AI-AI pair. Human stories are more dispersed: mean distance to human centroid is 22% greater than average AI radius.

Pipeline Architecture

  1. Structured narrative representations: Each story → JSON template along 10 NarraBench dimensions (Agent, Social Network, Event, Plot, Structure, Setting, Time, Revelation, Perspective, Style). GPT-5.1 extraction with zero-shot prompt + detailed JSON schema. Templates abstract away surface wording so downstream stages reason over narrative content, not style.

  2. Cross-source comparison: Discovery pool of 600 stories (100 prompts × 6 sources). For each prompt, all six templates presented to GPT-5.1 (high reasoning effort) → structured comparative analysis with per-source dimension notes, cross-source divergences, and executive summary.

  3. Feature discovery: 10 specialized expert prompts (one per NarraBench dimension) → GPT-5.1 proposes discriminative features as closed-form questions with discrete answer choices. Three discovery runs, union = 408 candidates, deduplicated via embedding clustering (F2LLM-4B, cosine threshold 0.85) → 304 features across 5 response types (124 categorical, 59 ordinal, 45 scale, 44 binary, 32 multi-select).

  4. Feature assignment: Gemini 3 Flash (minimal thinking) assigns all 304 features to all 61,608 stories. High reliability: Krippendorff's α = 0.88 across 5 independent runs, mean human-model Cohen's κ = 0.84.

  5. Classification: XGBoost + SHAP for feature importance decomposition. Bootstrap SHAP analysis (B=50 iterations, prompt-level resampling) assigns features to roles: 30 core (binary separation), 75 fingerprint (source-specific), rest excluded.

Costs and Scale

Limitations and Caveats