Adaptive Foundation Models for Online Decisions: HyperAgent with Fast Incremental Uncertainty Estimation
<|reference_start|>Adaptive Foundation Models for Online Decisions: HyperAgent with Fast Incremental Uncertainty Estimation: Foundation models often struggle with uncertainty when faced with new situations in online decision-making, necessitating scalable and efficient exploration to resolve this uncertainty. We introduce GPT-HyperAgent, an augmentation of GPT with HyperAgent for uncertainty-aware, scalable exploration in contextual bandits, a fundamental online decision problem involving natural language input. We prove that HyperAgent achieves fast incremental uncertainty estimation with $\tilde{O}(\log T)$ per-step computational complexity over $T$ periods under the linear realizable assumption. Our analysis demonstrates that HyperAgent's regret order matches that of exact Thompson sampling in linear contextual bandits, closing a significant theoretical gap in scalable exploration. Empirical results in real-world contextual bandit tasks, such as automated content moderation with human feedback, validate the practical effectiveness of GPT-HyperAgent for safety-critical decisions. Our code is open-sourced at \url{https://github.com/szrlee/GPT-HyperAgent/}.<|reference_end|>
@article{li2024adaptive,
    title={Adaptive Foundation Models for Online Decisions: HyperAgent with Fast
  Incremental Uncertainty Estimation},
    author={Yingru Li, Jiawei Xu, Zhi-Quan Luo},
    journal={arXiv preprint arXiv:2407.13195},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.13195},
    primaryClass={cs.LG cs.AI cs.HC cs.IT math.IT stat.ML}
}