Efficacy of Language Model Self-Play in Non-Zero-Sum Games
<|reference_start|>Efficacy of Language Model Self-Play in Non-Zero-Sum Games: Game-playing agents like AlphaGo have achieved superhuman performance through self-play, which is theoretically guaranteed to yield optimal policies in competitive games. However, most language tasks are partially or fully cooperative, so it is an open question whether techniques like self-play can effectively be used to improve language models. We empirically investigate this question in a negotiation game setting known as Deal or No Deal (DoND). Crucially, the objective in DoND can be modified to produce a fully cooperative game, a strictly competitive one, or anything in between. We finetune language models in self-play over multiple rounds of filtered behavior cloning in DoND for each of these objectives. Contrary to expectations, we find that language model self-play leads to significant performance gains in both cooperation and competition with humans, suggesting that self-play and related techniques have promise despite a lack of theoretical guarantees.<|reference_end|>
@article{liao2024efficacy,
    title={Efficacy of Language Model Self-Play in Non-Zero-Sum Games},
    author={Austen Liao, Nicholas Tomlin, Dan Klein},
    journal={arXiv preprint arXiv:2406.18872},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.18872},
    primaryClass={cs.CL}
}