  Offline reinforcement learning (RL) struggles in environments with rich and
noisy inputs, where the agent only has access to a fixed dataset without
environment interactions. Past works have proposed common workarounds based on
the pre-training of state representations, followed by policy training. In this
work, we introduce a simple, yet effective approach for learning state
representations. Our method, Behavior Prior Representation (BPR), learns state
representations with an easy-to-integrate objective based on behavior cloning
of the dataset: we first learn a state representation by mimicking actions from
the dataset, and then train a policy on top of the fixed representation, using
any off-the-shelf Offline RL algorithm. Theoretically, we prove that BPR
carries out performance guarantees when integrated into algorithms that have
either policy improvement guarantees (conservative algorithms) or produce lower
bounds of the policy values (pessimistic algorithms). Empirically, we show that
BPR combined with existing state-of-the-art Offline RL algorithms leads to
significant improvements across several offline control benchmarks. The code is
available at \url{https://github.com/bit1029public/offline_bpr}.

Machine Learning;Artificial Intelligence