Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learning
<|reference_start|>Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement Learning: We propose training fitted Q-iteration with log-loss (FQI-log) for batch reinforcement learning (RL). We show that the number of samples needed to learn a near-optimal policy with FQI-log scales with the accumulated cost of the optimal policy, which is zero in problems where acting optimally achieves the goal and incurs no cost. In doing so, we provide a general framework for proving small-cost bounds, i.e. bounds that scale with the optimal achievable cost, in batch RL. Moreover, we empirically verify that FQI-log uses fewer samples than FQI trained with squared loss on problems where the optimal policy reliably achieves the goal.<|reference_end|>
@article{ayoub2024switching,
    title={Switching the Loss Reduces the Cost in Batch (Offline) Reinforcement
  Learning},
    author={Alex Ayoub, Kaiwen Wang, Vincent Liu, Samuel Robertson, James
  McInerney, Dawen Liang, Nathan Kallus, and Csaba Szepesv'ari},
    journal={arXiv preprint arXiv:2403.05385},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.05385},
    primaryClass={cs.LG}
}