Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning
<|reference_start|>Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled Q-Learning: We present a novel method aimed at enhancing the sample efficiency of ensemble Q learning. Our proposed approach integrates multi-head self-attention into the ensembled Q networks while bootstrapping the state-action pairs ingested by the ensemble. This not only results in performance improvements over the original REDQ (Chen et al. 2021) and its variant DroQ (Hi-raoka et al. 2022), thereby enhancing Q predictions, but also effectively reduces both the average normalized bias and standard deviation of normalized bias within Q-function ensembles. Importantly, our method also performs well even in scenarios with a low update-to-data (UTD) ratio. Notably, the implementation of our proposed method is straightforward, requiring minimal modifications to the base model.<|reference_end|>
@article{khan2024smart,
    title={Smart Sampling: Self-Attention and Bootstrapping for Improved Ensembled
  Q-Learning},
    author={Muhammad Junaid Khan, Syed Hammad Ahmed, Gita Sukthankar},
    journal={arXiv preprint arXiv:2405.08252},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.08252},
    primaryClass={cs.LG cs.AI}
}