Roll the dice & look before you leap: Going beyond the creative limits
  of next-token prediction
We design a suite of minimal algorithmic tasks that are a loose abstraction
of open-ended real-world tasks. This allows us to cleanly and controllably
quantify the creative limits of the present-day language model. Much like
real-world tasks that require a creative, far-sighted leap of thought, our
tasks require an implicit, open-ended stochastic planning step that either (a)
discovers new connections in an abstract knowledge graph (like in wordplay,
drawing analogies, or research) or (b) constructs new patterns (like in
designing math problems or new proteins). In these tasks, we empirically and
conceptually argue how next-token learning is myopic and memorizes excessively;
comparatively, multi-token approaches, namely teacherless training and
diffusion models, excel in producing diverse and original output. Secondly, in
our tasks, we find that to elicit randomness from the Transformer without
hurting coherence, it is better to inject noise right at the input layer (via a
method we dub hash-conditioning) rather than defer to temperature sampling from
the output layer. Thus, our work offers a principled, minimal test-bed for
analyzing open-ended creative skills, and offers new arguments for going beyond
next-token learning and softmax-based sampling. We make part of the code
available under https://github.com/chenwu98/algorithmic-creativity
680826892b4aaa243886d2eb
Vaishnavh Nagarajan
extracted_pending
2025-06-03T12:10:23.915Z
683ee6255d1f6caeaf6ea96f
/avatars/a4337e0824915cbdfcad8da245962dc9.svg
Vaishnavh Nagarajan
user
vaishnavh94
680826892b4aaa243886d2ec
Chen Henry Wu
claimed_verified
2025-05-20T19:29:48.288Z
636577efe7a78348d820a4cc
https://cdn-avatars.huggingface.co/v1/production/uploads/1667594347446-636577efe7a78348d820a4cc.jpeg
Chen Wu
user
ChenWu98
680826892b4aaa243886d2ed
Charles Ding
680826892b4aaa243886d2ee
Aditi Raghunathan
Roll the dice & look before you leap: Going beyond the creative limits
  of next-token prediction
We design a suite of minimal algorithmic tasks that are a loose abstraction
of open-ended real-world tasks. This allows us to cleanly and controllably
quantify the creative limits of the present-day language model. Much like
real-world tasks that require a creative, far-sighted leap of thought, our
tasks require an implicit, open-ended stochastic planning step that either (a)
discovers new connections in an abstract knowledge graph (like in wordplay,
drawing analogies, or research) or (b) constructs new patterns (like in
designing math problems or new proteins). In these tasks, we empirically and
conceptually argue how next-token learning is myopic and memorizes excessively;
comparatively, multi-token approaches, namely teacherless training and
diffusion models, excel in producing diverse and original output. Secondly, in
our tasks, we find that to elicit randomness from the Transformer without
hurting coherence, it is better to inject noise right at the input layer (via a
method we dub hash-conditioning) rather than defer to temperature sampling from
the output layer. Thus, our work offers a principled, minimal test-bed for
analyzing open-ended creative skills, and offers new arguments for going beyond
next-token learning and softmax-based sampling. We make part of the code
available under https://github.com/chenwu98/algorithmic-creativity
Minimal algorithmic tasks reveal that multi-token approaches and hash-conditioning improve open-ended creativity in language models beyond next-token learning and softmax-based sampling.
Transformer hash-conditioning next-token learning diffusion models teacherless training