Answers.Body: <p>In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew.</p>
  <p>The point of regret as a metric therefore is to compare your choices with the ideal choices. This makes sense in MABs, because although the primary goal is to gain the most reward, the <em>learning</em> part of the goal is to calculate from experience what are the best choices - usually whilst sacrificing as little as possible in the process.</p>
  <p>The formula captures that concept, so does not concern itself with individual rewards in the past that could have been due to good or bad luck. Hence it uses expected (or mean) rewards.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-27T15:41:56.427Z
Answers.LastActivityDate: 2020-09-27T15:50:53.120Z
Answers.LastEditDate: 2020-09-27T15:50:53.120Z
Answers.UserDisplayName: Neil Slater
Body: <p>Consider a multi-armed bandit(MAB). There are <span class="math-container">$k$</span> arms, with reward distributions <span class="math-container">$R_i$</span> where <span class="math-container">$1 \leq i \leq k$</span>. Let <span class="math-container">$\mu_i$</span> denote the mean of the <span class="math-container">$i^{th}$</span> distribution.</p>
  <p>If we run the multi-armed bandit <em>experiment</em>  for <span class="math-container">$T$</span> rounds, the &quot;pseudo regret&quot; is defined as <span class="math-container">$$\text{Regret}_T = \sum_{t=1}^T \mu^* - \mu_{it},$$</span> where <span class="math-container">$\mu^*$</span> denotes the highest mean among all the <span class="math-container">$k$</span> distributions.</p>
  <p><strong>Why is regret defined like this?</strong> From what I understand, at time-step <span class="math-container">$t$</span>, the actual reward received is <span class="math-container">$r_t \sim R_{it} $</span> and not <span class="math-container">$\mu_{it}$</span> - so shouldn't that be a part of the expression for regret instead?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-26T17:13:04.843Z
Comments.Text: We don’t care what happens for one instance at one time step, we care about what happens in expectation at that time step, hence the mean
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-27T15:20:57.143Z
Comments.Text: Another reason is that, if we were to define regret with respect to the observed rewards, we might get *negative* regret whenever the agent just happens to be lucky.
Comments.UserDisplayName: Robby Goetschalckx
Tags: <reinforcement-learning><definitions><rewards><multi-armed-bandits><regret>
Title: Why is regret so defined in MABs?
UserDisplayName: stoic-santiago
ThreadText: # Why is regret so defined in MABs?
  
  (Asked by: username_0 on 2020-09-26)
  
  Consider a multi-armed bandit(MAB). There are $k$ arms, with reward distributions $R\_i$ where $1 \leq i \leq k$. Let $\mu\_i$ denote the mean of the $i^{th}$ distribution.
  
  If we run the multi-armed bandit *experiment* for $T$ rounds, the "pseudo regret" is defined as $$\text{Regret}\_T = \sum\_{t=1}^T \mu^\* - \mu\_{it},$$ where $\mu^\*$ denotes the highest mean among all the $k$ distributions.
  
  **Why is regret defined like this?** From what I understand, at time-step $t$, the actual reward received is $r\_t \sim R\_{it} $ and not $\mu\_{it}$ - so shouldn't that be a part of the expression for regret instead?
  
  **Tags:** reinforcement-learning, definitions, rewards, multi-armed-bandits, regret
  
  > **Comments:**
  > **username_1**: We don’t care what happens for one instance at one time step, we care about what happens in expectation at that time step, hence the mean  
  > **username_2**: Another reason is that, if we were to define regret with respect to the observed rewards, we might get *negative* regret whenever the agent just happens to be lucky.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-09-27. Score: 4 | Accepted answer
  
  In short, you don't regret your bad luck that you could do nothing about, you regret your bad choices that you could have done something about if only you knew.
  
  The point of regret as a metric therefore is to compare your choices with the ideal choices. This makes sense in MABs, because although the primary goal is to gain the most reward, the *learning* part of the goal is to calculate from experience what are the best choices - usually whilst sacrificing as little as possible in the process.
  
  The formula captures that concept, so does not concern itself with individual rewards in the past that could have been due to good or bad luck. Hence it uses expected (or mean) rewards.
  
  
  ---
  
  