Body: <p>Multi Armed Bandits (MABs) are a broad field of research pursuing different streams. In addition to the common objective of maximizing the cumulative reward, there are also so-called Best-Arm (identification) Bandits, cf. <a href="https://tor-lattimore.com/downloads/book/book.pdf" rel="nofollow noreferrer">Lattimore and Szepesvári (2020), Chapter 33</a> or <a href="http://certis.enpc.fr/%7Eaudibert/Mes%20articles/COLT10.pdf" rel="nofollow noreferrer">Audibert et al. (2010)</a>. The goal of the latter algorithms is to identify only the best arm (consequently, they aim to maximize the simple regret instead of the cumulative regret).</p>
  <p>Since a Best-Arm Bandit also involves a stochastic environment and also receives rewards by taking actions, <strong>I wonder if Best-Arm Bandits belong to the domain of reinforcement learning?</strong></p>
  <p>I came across this related <a href="https://ai.stackexchange.com/questions/20850/are-bandits-considered-an-rl-approach">post</a> (which, however, refers to Multi Armed Bandits in terms of the cumulative regret). From the responses in the post, my understanding is that the &quot;purely evaluative feedback&quot; should also apply to Best-Arm Bandits.</p>
  <p>My intention would categorize Best Armed Bandits as Reinforcement Learning, not only because of the presence of the evaluative feedback but also because they constitute the simplest form of all Reinforcement Learning problems at all, namely a Markov Decision Process with only one single state <span class="math-container">$s$</span>, a discrete set of actions <span class="math-container">$a\in A$</span> and a reward function <span class="math-container">$r(s,a)$</span> with the associated goal of maximizing the average reward. For an infinite horizon, this should be the same goal as identifying the best arm.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-17T14:20:45.710Z
Comments.Text: Can you provide a reference that describes this "Best-Arm (identification) Bandits" (so that we are all have your context)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-17T15:34:04.940Z
Comments.Text: I added two references regarding Best-Arm Bandits, thank you for pointing it out.
Comments.UserDisplayName: D. B.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-17T15:46:13.180Z
Comments.Text: The pure goal of identifying the "best arm" without other constraints is a pretty dull problem. Pull each arm millions of times, calculate the mean reward, the highest mean is the estimate for the "best arm". There must be some constraint or complication to make the subject noteworthy enough to be named. With MABs it is minimising regret whilst learning. For BABs, there must be something else - is it something you understand from the papers and can summarise in the question?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-17T15:50:04.580Z
Comments.Text: For example, are BAB's rated on an accuracy to identify the "best" vs number of learning steps so far, thus constrained by time and/or confidence measures?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-17T18:13:55.317Z
Comments.Text: No this is by no means a dull problem. 
  Not only the domain of Best Arm Bandits focuses on this type of problem but also other popular streams of research such as "Ranking and Selection". In practice, it is not possible to pull each arm, for example, 1 million times. This would is way too computationally expensive (in particular when there is a huge number of arms).
  
  
  Instead, often a certain computational budget (of iterations or simulations) is assumed, within which the best arm needs to be identified.
Comments.UserDisplayName: D. B.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-17T22:39:11.983Z
Comments.Text: Bandit problems are definitely a type of reinforcement learning, but I feel it would be strange to call an algorithm which solves a bandit problem a "reinforcement learning algorithm" for the same reason it would be strange to call a square a rectangle.
Comments.UserDisplayName: Taw
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-18T08:01:22.137Z
Comments.Text: Was about to answer, but then realised you don't give any motivation for categorising BAB's as reinforcement learning. Other than it is clearly a related subject, what is your purpose for making a categorisation? Is it so you can file your notes on the subject under a particular heading? Do you wish to use any particular result or theory from RL in order to solve a BAB problem?
Comments.UserDisplayName: Neil Slater
Tags: <reinforcement-learning><multi-armed-bandits>
Title: Are Best-Arm Bandits considered to be reinforcement learning?
UserDisplayName: D. B.
ThreadText: # Are Best-Arm Bandits considered to be reinforcement learning?
  
  (Asked by: username_0 on 2022-02-17)
  
  Multi Armed Bandits (MABs) are a broad field of research pursuing different streams. In addition to the common objective of maximizing the cumulative reward, there are also so-called Best-Arm (identification) Bandits, cf. [Lattimore and Szepesvári (2020), Chapter 33](https://tor-lattimore.com/downloads/book/book.pdf) or [Audibert et al. (2010)](http://certis.enpc.fr/%7Eaudibert/Mes%20articles/COLT10.pdf). The goal of the latter algorithms is to identify only the best arm (consequently, they aim to maximize the simple regret instead of the cumulative regret).
  
  Since a Best-Arm Bandit also involves a stochastic environment and also receives rewards by taking actions, **I wonder if Best-Arm Bandits belong to the domain of reinforcement learning?**
  
  I came across this related [post](https://ai.stackexchange.com/questions/20850/are-bandits-considered-an-rl-approach) (which, however, refers to Multi Armed Bandits in terms of the cumulative regret). From the responses in the post, my understanding is that the "purely evaluative feedback" should also apply to Best-Arm Bandits.
  
  My intention would categorize Best Armed Bandits as Reinforcement Learning, not only because of the presence of the evaluative feedback but also because they constitute the simplest form of all Reinforcement Learning problems at all, namely a Markov Decision Process with only one single state $s$, a discrete set of actions $a\in A$ and a reward function $r(s,a)$ with the associated goal of maximizing the average reward. For an infinite horizon, this should be the same goal as identifying the best arm.
  
  **Tags:** reinforcement-learning, multi-armed-bandits
  > **Comments:**
  > **username_1**: Can you provide a reference that describes this "Best-Arm (identification) Bandits" (so that we are all have your context)?  
  > **username_0**: I added two references regarding Best-Arm Bandits, thank you for pointing it out.  
  > **username_2**: The pure goal of identifying the "best arm" without other constraints is a pretty dull problem. Pull each arm millions of times, calculate the mean reward, the highest mean is the estimate for the "best arm". There must be some constraint or complication to make the subject noteworthy enough to be named. With MABs it is minimising regret whilst learning. For BABs, there must be something else - is it something you understand from the papers and can summarise in the question?  
  > **username_2**: For example, are BAB's rated on an accuracy to identify the "best" vs number of learning steps so far, thus constrained by time and/or confidence measures?  
  > **username_0**: No this is by no means a dull problem.  Not only the domain of Best Arm Bandits focuses on this type of problem but also other popular streams of research such as "Ranking and Selection". In practice, it is not possible to pull each arm, for example, 1 million times. This would is way too computationally expensive (in particular when there is a huge number of arms).   Instead, often a certain computational budget (of iterations or simulations) is assumed, within which the best arm needs to be identified.  
  > **username_3**: Bandit problems are definitely a type of reinforcement learning, but I feel it would be strange to call an algorithm which solves a bandit problem a "reinforcement learning algorithm" for the same reason it would be strange to call a square a rectangle.  
  > **username_2**: Was about to answer, but then realised you don't give any motivation for categorising BAB's as reinforcement learning. Other than it is clearly a related subject, what is your purpose for making a categorisation? Is it so you can file your notes on the subject under a particular heading? Do you wish to use any particular result or theory from RL in order to solve a BAB problem?  
  
  ---
  
  ## 0 answers
  
  