Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
<|reference_start|>Offline Reinforcement Learning from Datasets with Structured Non-Stationarity: Current Reinforcement Learning (RL) is often limited by the large amount of data needed to learn a successful policy. Offline RL aims to solve this issue by using transitions collected by a different behavior policy. We address a novel Offline RL problem setting in which, while collecting the dataset, the transition and reward functions gradually change between episodes but stay constant within each episode. We propose a method based on Contrastive Predictive Coding that identifies this non-stationarity in the offline dataset, accounts for it when training a policy, and predicts it during evaluation. We analyze our proposed method and show that it performs well in simple continuous control tasks and challenging, high-dimensional locomotion tasks. We show that our method often achieves the oracle performance and performs better than baselines.<|reference_end|>
@article{ackermann2024offline,
    title={Offline Reinforcement Learning from Datasets with Structured
  Non-Stationarity},
    author={Johannes Ackermann, Takayuki Osa, Masashi Sugiyama},
    journal={arXiv preprint arXiv:2405.14114},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.14114},
    primaryClass={cs.LG cs.AI}
}