Strategizing against Q-learners: A Control-theoretical Approach
<|reference_start|>Strategizing against Q-learners: A Control-theoretical Approach: In this paper, we explore the susceptibility of the independent Q-learning algorithms (a classical and widely used multi-agent reinforcement learning method) to strategic manipulation of sophisticated opponents in normal-form games played repeatedly. We quantify how much strategically sophisticated agents can exploit naive Q-learners if they know the opponents' Q-learning algorithm. To this end, we formulate the strategic actors' interactions as a stochastic game (whose state encompasses Q-function estimates of the Q-learners) as if the Q-learning algorithms are the underlying dynamical system. We also present a quantization-based approximation scheme to tackle the continuum state space and analyze its performance for two competing strategic actors and a single strategic actor both analytically and numerically.<|reference_end|>
@article{arslantas2024strategizing,
    title={Strategizing against Q-learners: A Control-theoretical Approach},
    author={Yuksel Arslantas, Ege Yuceel and Muhammed O. Sayin},
    journal={IEEE Control Systems Letters 8 (2024) 1733-1738},
    year={2024},
    doi={10.1109/LCSYS.2024.3416240},
    archivePrefix={arXiv},
    eprint={2403.08906},
    primaryClass={cs.GT cs.AI math.OC}
}