title: Frankentext: Stitching random text fragments into long-form narratives
summary: We introduce Frankentexts, a new type of long-form narratives produced by
  LLMs under the extreme constraint that most tokens (e.g., 90%) must be copied
  verbatim from human writings. This task presents a challenging test of
  controllable generation, requiring models to satisfy a writing prompt,
  integrate disparate text fragments, and still produce a coherent narrative. To
  generate Frankentexts, we instruct the model to produce a draft by selecting
  and combining human-written passages, then iteratively revise the draft while
  maintaining a user-specified copy ratio. We evaluate the resulting Frankentexts
  along three axes: writing quality, instruction adherence, and detectability.
  Gemini-2.5-Pro performs surprisingly well on this task: 81% of its Frankentexts
  are coherent and 100% relevant to the prompt. Notably, up to 59% of these
  outputs are misclassified as human-written by detectors like Pangram, revealing
  limitations in AI text detectors. Human annotators can sometimes identify
  Frankentexts through their abrupt tone shifts and inconsistent grammar between
  segments, especially in longer generations. Beyond presenting a challenging
  generation task, Frankentexts invite discussion on building effective detectors
  for this new grey zone of authorship, provide training data for mixed
  authorship detection, and serve as a sandbox for studying human-AI co-writing
  processes.
paper_authors._id: 683f107dfde42c09123a1ea5
paper_authors.name: Chau Minh Pham
paper_authors._id: 683f107dfde42c09123a1ea6
paper_authors.name: Jenna Russell
paper_authors._id: 683f107dfde42c09123a1ea7
paper_authors.name: Dzung Pham
paper_authors._id: 683f107dfde42c09123a1ea8
paper_authors.name: Mohit Iyyer
paper_title: Frankentext: Stitching random text fragments into long-form narratives
paper_summary: We introduce Frankentexts, a new type of long-form narratives produced by
  LLMs under the extreme constraint that most tokens (e.g., 90%) must be copied
  verbatim from human writings. This task presents a challenging test of
  controllable generation, requiring models to satisfy a writing prompt,
  integrate disparate text fragments, and still produce a coherent narrative. To
  generate Frankentexts, we instruct the model to produce a draft by selecting
  and combining human-written passages, then iteratively revise the draft while
  maintaining a user-specified copy ratio. We evaluate the resulting Frankentexts
  along three axes: writing quality, instruction adherence, and detectability.
  Gemini-2.5-Pro performs surprisingly well on this task: 81% of its Frankentexts
  are coherent and 100% relevant to the prompt. Notably, up to 59% of these
  outputs are misclassified as human-written by detectors like Pangram, revealing
  limitations in AI text detectors. Human annotators can sometimes identify
  Frankentexts through their abrupt tone shifts and inconsistent grammar between
  segments, especially in longer generations. Beyond presenting a challenging
  generation task, Frankentexts invite discussion on building effective detectors
  for this new grey zone of authorship, provide training data for mixed
  authorship detection, and serve as a sandbox for studying human-AI co-writing
  processes.
paper_ai_summary: Frankentexts challenge LLMs to create coherent long-form narratives using mostly copied human-written text, highlighting limitations in AI text detectors and offering insights into human-AI collaboration.
paper_ai_keywords: LLMs Frankentexts controllable generation writing quality instruction adherence detectability Gemini-2.5-Pro Pangram mixed authorship detection human-AI co-writing processes