  Offline reinforcement learning (RL) faces a significant challenge of
distribution shift. Model-free offline RL penalizes the Q value for
out-of-distribution (OOD) data or constrains the policy closed to the behavior
policy to tackle this problem, but this inhibits the exploration of the OOD
region. Model-based offline RL, which uses the trained environment model to
generate more OOD data and performs conservative policy optimization within
that model, has become an effective method for this problem. However, the
current model-based algorithms rarely consider agent robustness when
incorporating conservatism into policy. Therefore, the new model-based offline
algorithm with a conservative Bellman operator (MICRO) is proposed. This method
trades off performance and robustness via introducing the robust Bellman
operator into the algorithm. Compared with previous model-based algorithms with
robust adversarial models, MICRO can significantly reduce the computation cost
by only choosing the minimal Q value in the state uncertainty set. Extensive
experiments demonstrate that MICRO outperforms prior RL algorithms in offline
RL benchmark and is considerably robust to adversarial perturbations.

Machine Learning;Artificial Intelligence