The Virtues of Pessimism in Inverse Reinforcement Learning
<|reference_start|>The Virtues of Pessimism in Inverse Reinforcement Learning: Inverse Reinforcement Learning (IRL) is a powerful framework for learning complex behaviors from expert demonstrations. However, it traditionally requires repeatedly solving a computationally expensive reinforcement learning (RL) problem in its inner loop. It is desirable to reduce the exploration burden by leveraging expert demonstrations in the inner-loop RL. As an example, recent work resets the learner to expert states in order to inform the learner of high-reward expert states. However, such an approach is infeasible in the real world. In this work, we consider an alternative approach to speeding up the RL subroutine in IRL: \emph{pessimism}, i.e., staying close to the expert's data distribution, instantiated via the use of offline RL algorithms. We formalize a connection between offline RL and IRL, enabling us to use an arbitrary offline RL algorithm to improve the sample efficiency of IRL. We validate our theory experimentally by demonstrating a strong correlation between the efficacy of an offline RL algorithm and how well it works as part of an IRL procedure. By using a strong offline RL algorithm as part of an IRL procedure, we are able to find policies that match expert performance significantly more efficiently than the prior art.<|reference_end|>
@article{wu2024the,
    title={The Virtues of Pessimism in Inverse Reinforcement Learning},
    author={David Wu and Gokul Swamy and J. Andrew Bagnell and Zhiwei Steven Wu
  and Sanjiban Choudhury},
    journal={arXiv preprint arXiv:2402.02616},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.02616},
    primaryClass={cs.LG}
}