Learning Goal-Conditioned Policies from Sub-Optimal Offline Data via Metric Learning
<|reference_start|>Learning Goal-Conditioned Policies from Sub-Optimal Offline Data via Metric Learning: We address the problem of learning optimal behavior from sub-optimal datasets for goal-conditioned offline reinforcement learning. To do so, we propose the use of metric learning to approximate the optimal value function for goal-conditioned offline RL problems under sparse rewards, invertible actions and deterministic transitions. We introduce distance monotonicity, a property for representations to recover optimality and propose an optimization objective that leads to such property. We use the proposed value function to guide the learning of a policy in an actor-critic fashion, a method we name MetricRL. Experimentally, we show that our method estimates optimal behaviors from severely sub-optimal offline datasets without suffering from out-of-distribution estimation errors. We demonstrate that MetricRL consistently outperforms prior state-of-the-art goal-conditioned RL methods in learning optimal policies from sub-optimal offline datasets.<|reference_end|>
@article{reichlin2024learning,
    title={Learning Goal-Conditioned Policies from Sub-Optimal Offline Data via
  Metric Learning},
    author={Alfredo Reichlin, Miguel Vasco, Hang Yin, Danica Kragic},
    journal={arXiv preprint arXiv:2402.10820},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.10820},
    primaryClass={cs.LG}
}