title: Efficient Exploration for LLMs
summary: We present evidence of substantial benefit from efficient exploration in
  gathering human feedback to improve large language models. In our experiments,
  an agent sequentially generates queries while fitting a reward model to the
  feedback received. Our best-performing agent generates queries using double
  Thompson sampling, with uncertainty represented by an epistemic neural network.
  Our results demonstrate that efficient exploration enables high levels of
  performance with far fewer queries. Further, both uncertainty estimation and
  the choice of exploration scheme play critical roles.
paper_authors._id: 65bc65ce259bc6caeba86ac1
paper_authors.name: Vikranth Dwaracherla
paper_authors._id: 65bc65ce259bc6caeba86ac2
paper_authors.name: Seyed Mohammad Asghari
paper_authors._id: 65bc65ce259bc6caeba86ac3
paper_authors.name: Botao Hao
paper_authors._id: 65bc65ce259bc6caeba86ac4
paper_authors.name: Benjamin Van Roy
paper_title: Efficient Exploration for LLMs
paper_summary: We present evidence of substantial benefit from efficient exploration in
  gathering human feedback to improve large language models. In our experiments,
  an agent sequentially generates queries while fitting a reward model to the
  feedback received. Our best-performing agent generates queries using double
  Thompson sampling, with uncertainty represented by an epistemic neural network.
  Our results demonstrate that efficient exploration enables high levels of
  performance with far fewer queries. Further, both uncertainty estimation and
  the choice of exploration scheme play critical roles.
paper_ai_summary: Efficient exploration using double Thompson sampling and epistemic neural network uncertainty estimation significantly improves large language models with reduced query numbers.
paper_ai_keywords: double Thompson sampling epistemic neural network reward model