Goals as Reward-Producing Programs
<|reference_start|>Goals as Reward-Producing Programs: People are remarkably capable of generating their own goals, beginning with child's play and continuing into adulthood. Despite considerable empirical and computational work on goals and goal-oriented behavior, models are still far from capturing the richness of everyday human goals. Here, we bridge this gap by collecting a dataset of human-generated playful goals (in the form of scorable, single-player games), modeling them as reward-producing programs, and generating novel human-like goals through program synthesis. Reward-producing programs capture the rich semantics of goals through symbolic operations that compose, add temporal constraints, and allow for program execution on behavioral traces to evaluate progress. To build a generative model of goals, we learn a fitness function over the infinite set of possible goal programs and sample novel goals with a quality-diversity algorithm. Human evaluators found that model-generated goals, when sampled from partitions of program space occupied by human examples, were indistinguishable from human-created games. We also discovered that our model's internal fitness scores predict games that are evaluated as more fun to play and more human-like.<|reference_end|>
@article{davidson2024goals,
    title={Goals as Reward-Producing Programs},
    author={Guy Davidson, Graham Todd, Julian Togelius, Todd M. Gureckis, Brenden
  M. Lake},
    journal={arXiv preprint arXiv:2405.13242},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.13242},
    primaryClass={cs.AI}
}