Answers.Body: <p>If you changed this:</p>
  <blockquote>
  <p>&quot;Multi-armed-bandit selects an action based on the future reward&quot;</p>
  </blockquote>
  <p>To <em>this</em></p>
  <blockquote>
  <p>&quot;Multi-armed-bandit selects an action based on the <em>expected</em> future reward&quot;</p>
  </blockquote>
  <p>The description would be better in my opinion. Perhaps <em>expected</em> is just missing on the slide for brevity. I would rate the original quote as <em>mostly true</em> and give the author the benefit of the doubt here.</p>
  <p>However, the quote is out of context, and how a multi-armed bandit agent selects an action can be more complex than simply considering the expected future reward. In bandit agents, the agent is usually considered to be learning &quot;live&quot;, and has to balance exploring for better options versus exploiting a known best option so far. This is commonly referred to as <em>minimising regret</em> - which technically is a measure of how the agent performs <em>in expectation</em> (using actual action choices, but mean rewards for those actions as opposed to observed rewards) against an imaginary version of itself that always chose the best action in hindsight (or by revealing hidden knowledge of true values in the end for test cases).</p>
  <p>If you include this training behaviour in the description, then you could also say:</p>
  <blockquote>
  <p>&quot;Multi-armed-bandit selects an action based on minimising regret&quot;</p>
  </blockquote>
  <p>Although this is not technically excluding basing on expected future reward, since usually the estimates of action values are used to make a decision, and &quot;based on&quot; is not the same as &quot;<strong>only</strong> considering&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-07T15:52:43.337Z
Answers.LastActivityDate: 2023-06-07T15:58:27.330Z
Answers.LastEditDate: 2023-06-07T15:58:27.330Z
Answers.UserDisplayName: Neil Slater
Body: <p>During my machine learning lecture I saw the following statement on the slides: &quot;Multi-armed-bandit selects an action based on the future reward&quot;</p>
  <p>Is that statement true?</p>
  <p>In my opinion it is false because the selection of the action is not based on the reward directly as it is not known but on some strategy to maximize the reward in the long run.</p>
  <p>Could maybe someone provide some references if it true or not, as I could not find any.</p>
  
Tags: <reinforcement-learning><multi-armed-bandits>
Title: Is it true that a multi-armed-bandit selects an action based on the future reward?
UserDisplayName: max0r
ThreadText: # Is it true that a multi-armed-bandit selects an action based on the future reward?
  
  (Asked by: username_0 on 2023-06-07)
  
  During my machine learning lecture I saw the following statement on the slides: "Multi-armed-bandit selects an action based on the future reward"
  
  Is that statement true?
  
  In my opinion it is false because the selection of the action is not based on the reward directly as it is not known but on some strategy to maximize the reward in the long run.
  
  Could maybe someone provide some references if it true or not, as I could not find any.
  
  **Tags:** reinforcement-learning, multi-armed-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-06-07. Score: 2 | Accepted answer
  
  If you changed this:
  
  > "Multi-armed-bandit selects an action based on the future reward"
  
  To *this*
  
  > "Multi-armed-bandit selects an action based on the *expected* future reward"
  
  The description would be better in my opinion. Perhaps *expected* is just missing on the slide for brevity. I would rate the original quote as *mostly true* and give the author the benefit of the doubt here.
  
  However, the quote is out of context, and how a multi-armed bandit agent selects an action can be more complex than simply considering the expected future reward. In bandit agents, the agent is usually considered to be learning "live", and has to balance exploring for better options versus exploiting a known best option so far. This is commonly referred to as *minimising regret* - which technically is a measure of how the agent performs *in expectation* (using actual action choices, but mean rewards for those actions as opposed to observed rewards) against an imaginary version of itself that always chose the best action in hindsight (or by revealing hidden knowledge of true values in the end for test cases).
  
  If you include this training behaviour in the description, then you could also say:
  
  > "Multi-armed-bandit selects an action based on minimising regret"
  
  Although this is not technically excluding basing on expected future reward, since usually the estimates of action values are used to make a decision, and "based on" is not the same as "**only** considering".
  
  
  ---
  
  