Modifying Large Language Model Post-Training for Diverse Creative
  Writing
As creative writing tasks do not have singular correct answers, large
language models (LLMs) trained to perform these tasks should be able to
generate diverse valid outputs. However, LLM post-training often focuses on
improving generation quality but neglects to facilitate output diversity.
Hence, in creative writing generation, we investigate post-training approaches
to promote both output diversity and quality. Our core idea is to include
deviation -- the degree of difference between a training sample and all other
samples with the same prompt -- in the training objective to facilitate
learning from rare high-quality instances. By adopting our approach to direct
preference optimization (DPO) and odds ratio preference optimization (ORPO), we
demonstrate that we can promote the output diversity of trained models while
minimally decreasing quality. Our best model with 8B parameters could achieve
on-par diversity as a human-created dataset while having output quality similar
to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We
further validate our approaches with a human evaluation, an ablation, and a
comparison to an existing diversification approach, DivPO.
67e0beb474fc794321fb4ad7
John Joon Young Chung
admin_assigned
2025-03-24T12:42:42.640Z
630c92f6c169245d78fcfa19
/avatars/a4dd46b1b61f43530a1c934de29420af.svg
John Joon Young Chung
user
johnr0
67e0beb474fc794321fb4ad8
Vishakh Padmakumar
admin_assigned
2025-03-24T12:42:49.425Z
62a96c134a8579cc0be1eb16
/avatars/d1862985db7769f4d5e4ab82ecdc6796.svg
Vishakh Padmakumar
user
vishakhpk
67e0beb474fc794321fb4ad9
Melissa Roemmele
admin_assigned
2025-03-24T12:42:55.929Z
63a0ef9945edac9f75042337
https://cdn-avatars.huggingface.co/v1/production/uploads/1671491435912-noauth.jpeg
Melissa Roemmele
user
roemmele
67e0beb474fc794321fb4ada
Yuqian Sun
67e0beb474fc794321fb4adb
Max Kreminski
Modifying Large Language Model Post-Training for Diverse Creative
  Writing
As creative writing tasks do not have singular correct answers, large
language models (LLMs) trained to perform these tasks should be able to
generate diverse valid outputs. However, LLM post-training often focuses on
improving generation quality but neglects to facilitate output diversity.
Hence, in creative writing generation, we investigate post-training approaches
to promote both output diversity and quality. Our core idea is to include
deviation -- the degree of difference between a training sample and all other
samples with the same prompt -- in the training objective to facilitate
learning from rare high-quality instances. By adopting our approach to direct
preference optimization (DPO) and odds ratio preference optimization (ORPO), we
demonstrate that we can promote the output diversity of trained models while
minimally decreasing quality. Our best model with 8B parameters could achieve
on-par diversity as a human-created dataset while having output quality similar
to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We
further validate our approaches with a human evaluation, an ablation, and a
comparison to an existing diversification approach, DivPO.
Post-training approaches incorporating deviation into direct preference optimization and odds ratio preference optimization enhance the output diversity of large language models without significantly compromising quality.
LLMs direct preference optimization (DPO) odds ratio preference optimization (ORPO) deviation human-created dataset DivPO instruction-tuned models GPT-4o DeepSeek-R1