title: Modifying Large Language Model Post-Training for Diverse Creative
    Writing
summary: As creative writing tasks do not have singular correct answers, large
  language models (LLMs) trained to perform these tasks should be able to
  generate diverse valid outputs. However, LLM post-training often focuses on
  improving generation quality but neglects to facilitate output diversity.
  Hence, in creative writing generation, we investigate post-training approaches
  to promote both output diversity and quality. Our core idea is to include
  deviation -- the degree of difference between a training sample and all other
  samples with the same prompt -- in the training objective to facilitate
  learning from rare high-quality instances. By adopting our approach to direct
  preference optimization (DPO) and odds ratio preference optimization (ORPO), we
  demonstrate that we can promote the output diversity of trained models while
  minimally decreasing quality. Our best model with 8B parameters could achieve
  on-par diversity as a human-created dataset while having output quality similar
  to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We
  further validate our approaches with a human evaluation, an ablation, and a
  comparison to an existing diversification approach, DivPO.
paper_authors._id: 67e0beb474fc794321fb4ad7
paper_authors.name: John Joon Young Chung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:42:42.640Z
paper_authors._id: 630c92f6c169245d78fcfa19
paper_authors.avatarUrl: /avatars/a4dd46b1b61f43530a1c934de29420af.svg
paper_authors.fullname: John Joon Young Chung
paper_authors.type: user
paper_authors.user: johnr0
paper_authors._id: 67e0beb474fc794321fb4ad8
paper_authors.name: Vishakh Padmakumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:42:49.425Z
paper_authors._id: 62a96c134a8579cc0be1eb16
paper_authors.avatarUrl: /avatars/d1862985db7769f4d5e4ab82ecdc6796.svg
paper_authors.fullname: Vishakh Padmakumar
paper_authors.type: user
paper_authors.user: vishakhpk
paper_authors._id: 67e0beb474fc794321fb4ad9
paper_authors.name: Melissa Roemmele
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:42:55.929Z
paper_authors._id: 63a0ef9945edac9f75042337
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671491435912-noauth.jpeg
paper_authors.fullname: Melissa Roemmele
paper_authors.type: user
paper_authors.user: roemmele
paper_authors._id: 67e0beb474fc794321fb4ada
paper_authors.name: Yuqian Sun
paper_authors._id: 67e0beb474fc794321fb4adb
paper_authors.name: Max Kreminski
paper_title: Modifying Large Language Model Post-Training for Diverse Creative
    Writing
paper_summary: As creative writing tasks do not have singular correct answers, large
  language models (LLMs) trained to perform these tasks should be able to
  generate diverse valid outputs. However, LLM post-training often focuses on
  improving generation quality but neglects to facilitate output diversity.
  Hence, in creative writing generation, we investigate post-training approaches
  to promote both output diversity and quality. Our core idea is to include
  deviation -- the degree of difference between a training sample and all other
  samples with the same prompt -- in the training objective to facilitate
  learning from rare high-quality instances. By adopting our approach to direct
  preference optimization (DPO) and odds ratio preference optimization (ORPO), we
  demonstrate that we can promote the output diversity of trained models while
  minimally decreasing quality. Our best model with 8B parameters could achieve
  on-par diversity as a human-created dataset while having output quality similar
  to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We
  further validate our approaches with a human evaluation, an ablation, and a
  comparison to an existing diversification approach, DivPO.
paper_ai_summary: Post-training approaches incorporating deviation into direct preference optimization and odds ratio preference optimization enhance the output diversity of large language models without significantly compromising quality.
paper_ai_keywords: LLMs direct preference optimization (DPO) odds ratio preference optimization (ORPO) deviation human-created dataset DivPO instruction-tuned models GPT-4o DeepSeek-R1