Variable-Agnostic Causal Exploration for Reinforcement Learning
<|reference_start|>Variable-Agnostic Causal Exploration for Reinforcement Learning: Modern reinforcement learning (RL) struggles to capture real-world cause-and-effect dynamics, leading to inefficient exploration due to extensive trial-and-error actions. While recent efforts to improve agent exploration have leveraged causal discovery, they often make unrealistic assumptions of causal variables in the environments. In this paper, we introduce a novel framework, Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL), incorporating causal relationships to drive exploration in RL without specifying environmental causal variables. Our approach automatically identifies crucial observation-action steps associated with key variables using attention mechanisms. Subsequently, it constructs the causal graph connecting these steps, which guides the agent towards observation-action pairs with greater causal influence on task completion. This can be leveraged to generate intrinsic rewards or establish a hierarchy of subgoals to enhance exploration efficiency. Experimental results showcase a significant improvement in agent performance in grid-world, 2d games and robotic domains, particularly in scenarios with sparse rewards and noisy actions, such as the notorious Noisy-TV environments.<|reference_end|>
@article{nguyen2024variable-agnostic,
    title={Variable-Agnostic Causal Exploration for Reinforcement Learning},
    author={Minh Hoang Nguyen, Hung Le, Svetha Venkatesh},
    journal={arXiv preprint arXiv:2407.12437},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12437},
    primaryClass={cs.LG cs.AI}
}