title: Roll the dice & look before you leap: Going beyond the creative limits
    of next-token prediction
summary: We design a suite of minimal algorithmic tasks that are a loose abstraction
  of open-ended real-world tasks. This allows us to cleanly and controllably
  quantify the creative limits of the present-day language model. Much like
  real-world tasks that require a creative, far-sighted leap of thought, our
  tasks require an implicit, open-ended stochastic planning step that either (a)
  discovers new connections in an abstract knowledge graph (like in wordplay,
  drawing analogies, or research) or (b) constructs new patterns (like in
  designing math problems or new proteins). In these tasks, we empirically and
  conceptually argue how next-token learning is myopic and memorizes excessively;
  comparatively, multi-token approaches, namely teacherless training and
  diffusion models, excel in producing diverse and original output. Secondly, in
  our tasks, we find that to elicit randomness from the Transformer without
  hurting coherence, it is better to inject noise right at the input layer (via a
  method we dub hash-conditioning) rather than defer to temperature sampling from
  the output layer. Thus, our work offers a principled, minimal test-bed for
  analyzing open-ended creative skills, and offers new arguments for going beyond
  next-token learning and softmax-based sampling. We make part of the code
  available under https://github.com/chenwu98/algorithmic-creativity
paper_authors._id: 680826892b4aaa243886d2eb
paper_authors.name: Vaishnavh Nagarajan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-06-03T12:10:23.915Z
paper_authors._id: 683ee6255d1f6caeaf6ea96f
paper_authors.avatarUrl: /avatars/a4337e0824915cbdfcad8da245962dc9.svg
paper_authors.fullname: Vaishnavh Nagarajan
paper_authors.type: user
paper_authors.user: vaishnavh94
paper_authors._id: 680826892b4aaa243886d2ec
paper_authors.name: Chen Henry Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T19:29:48.288Z
paper_authors._id: 636577efe7a78348d820a4cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667594347446-636577efe7a78348d820a4cc.jpeg
paper_authors.fullname: Chen Wu
paper_authors.type: user
paper_authors.user: ChenWu98
paper_authors._id: 680826892b4aaa243886d2ed
paper_authors.name: Charles Ding
paper_authors._id: 680826892b4aaa243886d2ee
paper_authors.name: Aditi Raghunathan
paper_title: Roll the dice & look before you leap: Going beyond the creative limits
    of next-token prediction
paper_summary: We design a suite of minimal algorithmic tasks that are a loose abstraction
  of open-ended real-world tasks. This allows us to cleanly and controllably
  quantify the creative limits of the present-day language model. Much like
  real-world tasks that require a creative, far-sighted leap of thought, our
  tasks require an implicit, open-ended stochastic planning step that either (a)
  discovers new connections in an abstract knowledge graph (like in wordplay,
  drawing analogies, or research) or (b) constructs new patterns (like in
  designing math problems or new proteins). In these tasks, we empirically and
  conceptually argue how next-token learning is myopic and memorizes excessively;
  comparatively, multi-token approaches, namely teacherless training and
  diffusion models, excel in producing diverse and original output. Secondly, in
  our tasks, we find that to elicit randomness from the Transformer without
  hurting coherence, it is better to inject noise right at the input layer (via a
  method we dub hash-conditioning) rather than defer to temperature sampling from
  the output layer. Thus, our work offers a principled, minimal test-bed for
  analyzing open-ended creative skills, and offers new arguments for going beyond
  next-token learning and softmax-based sampling. We make part of the code
  available under https://github.com/chenwu98/algorithmic-creativity
paper_ai_summary: Minimal algorithmic tasks reveal that multi-token approaches and hash-conditioning improve open-ended creativity in language models beyond next-token learning and softmax-based sampling.
paper_ai_keywords: Transformer hash-conditioning next-token learning diffusion models teacherless training