Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction
<|reference_start|>Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction: In the face of difficult exploration problems in reinforcement learning, we study whether giving an agent an object-centric mapping (describing a set of items and their attributes) allow for more efficient learning. We found this problem is best solved hierarchically by modelling items at a higher level of state abstraction to pixels, and attribute change at a higher level of temporal abstraction to primitive actions. This abstraction simplifies the transition dynamic by making specific future states easier to predict. We make use of this to propose a fully model-based algorithm that learns a discriminative world model, plans to explore efficiently with only a count-based intrinsic reward, and can subsequently plan to reach any discovered (abstract) states.   We demonstrate the model's ability to (i) efficiently solve single tasks, (ii) transfer zero-shot and few-shot across item types and environments, and (iii) plan across long horizons. Across a suite of 2D crafting and MiniHack environments, we empirically show our model significantly out-performs state-of-the-art low-level methods (without abstraction), as well as performant model-free and model-based methods using the same abstraction. Finally, we show how to reinforce learn low level object-perturbing policies, as well as supervise learn the object mapping itself.<|reference_end|>
@article{gx-chen2024efficient,
    title={Efficient Exploration and Discriminative World Model Learning with an
  Object-Centric Abstraction},
    author={Anthony GX-Chen, Kenneth Marino, Rob Fergus},
    journal={arXiv preprint arXiv:2408.11816},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.11816},
    primaryClass={cs.LG cs.AI}
}