Predicting vs Acting: A Trade-off Between World Modeling & Agent Modeling
<|reference_start|>Predicting vs Acting: A Trade-off Between World Modeling & Agent Modeling: RLHF-aligned LMs have shown unprecedented ability on both benchmarks and long-form text generation, yet they struggle with one foundational task: next-token prediction. As RLHF models become agent models aimed at interacting with humans, they seem to lose their world modeling -- the ability to predict what comes next in arbitrary documents, which is the foundational training objective of the Base LMs that RLHF adapts.   Besides empirically demonstrating this trade-off, we propose a potential explanation: to perform coherent long-form generation, RLHF models restrict randomness via implicit blueprints. In particular, RLHF models concentrate probability on sets of anchor spans that co-occur across multiple generations for the same prompt, serving as textual scaffolding but also limiting a model's ability to generate documents that do not include these spans. We study this trade-off on the most effective current agent models, those aligned with RLHF, while exploring why this may remain a fundamental trade-off between models that act and those that predict, even as alignment techniques improve.<|reference_end|>
@article{li2024predicting,
    title={Predicting vs. Acting: A Trade-off Between World Modeling & Agent
  Modeling},
    author={Margaret Li, Weijia Shi, Artidoro Pagnoni, Peter West, Ari Holtzman},
    journal={arXiv preprint arXiv:2407.02446},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.02446},
    primaryClass={cs.CL cs.AI}
}