Do LLM Agents Have Regret? A Case Study in Online Learning and Games
<|reference_start|>Do LLM Agents Have Regret? A Case Study in Online Learning and Games: Large language models (LLMs) have been increasingly employed for (interactive) decision-making, via the development of LLM-based autonomous agents. Despite their emerging successes, the performance of LLM agents in decision-making has not been fully investigated through quantitative metrics, especially in the multi-agent setting when they interact with each other, a typical scenario in real-world LLM-agent applications. To better understand the limits of LLM agents in these interactive environments, we propose to study their interactions in benchmark decision-making settings in online learning and game theory, through the performance metric of \emph{regret}. We first empirically study the {no-regret} behaviors of LLMs in canonical (non-stationary) online learning problems, as well as the emergence of equilibria when LLM agents interact through playing repeated games. We then provide some theoretical insights into the no-regret behaviors of LLM agents, under certain assumptions on the supervised pre-training and the rationality model of human decision-makers who generate the data. Notably, we also identify (simple) cases where advanced LLMs such as GPT-4 fail to be no-regret. To promote the no-regret behaviors, we propose a novel \emph{unsupervised} training loss of \emph{regret-loss}, which, in contrast to the supervised pre-training loss, does not require the labels of (optimal) actions. We then establish the statistical guarantee of generalization bound for regret-loss minimization, followed by the optimization guarantee that minimizing such a loss may automatically lead to known no-regret learning algorithms. Our further experiments demonstrate the effectiveness of our regret-loss, especially in addressing the above ``regrettable'' cases.<|reference_end|>
@article{park2024do,
    title={Do LLM Agents Have Regret? A Case Study in Online Learning and Games},
    author={Chanwoo Park, Xiangyu Liu, Asuman Ozdaglar, Kaiqing Zhang},
    journal={arXiv preprint arXiv:2403.16843},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.16843},
    primaryClass={cs.LG cs.AI cs.GT}
}