Answers.Body: <p>The weighted average stands for a linear combination of all values, such that the sum of all weights is 1.</p>
  
  <p>More specifically, if you denote the rewards by a vector <span class="math-container">$X$</span>, the weighted average will be taking the dot product between <span class="math-container">$X$</span> and a vector <span class="math-container">$W$</span> such that <span class="math-container">$0 \le W_i \le 1$</span> and the sum of all <span class="math-container">$W_i$</span> is 1.</p>
  
  <p>If each <span class="math-container">$W_i = 1/n$</span> it will be a weighted average (a.k.a the mean). Using the exponential decay <span class="math-container">$W_i = \alpha^i/ (\sum W_i)$</span> is also a weighted average.</p>
  
  <p>Then both strategies to compute the Q value use weighted average of previous reward.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-01-19T00:18:16.470Z
Answers.LastActivityDate: 2019-10-05T17:33:20.577Z
Answers.LastEditDate: 2019-10-05T17:33:20.577Z
Answers.UserDisplayName: Marcelo Fornet
Body: <p>In the book <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction</a> (page 25), by Richard S. Sutton and Andrew G. Barto, there is a discussion of the k-armed bandit problem, where the expected reward from the bandits changes slightly over time (that is, the problem is non-stationary). Instead of updating the Q values by taking an average of all rewards, the book suggests using a constant step-size parameter, so as to give greater weight to more recent rewards. Thus:</p>
  
  <p><span class="math-container">$$ Q_{n+1} = Q_n + \alpha (R_n - Q_n),$$</span></p>
  
  <p>where <span class="math-container">$\alpha$</span> is a constant between 0 and 1. </p>
  
  <p>The book then states that <em>this a weighted average because the sum of the weights is equal to 1</em>. What does this mean? Why is this true?</p>
  
Tags: <machine-learning><reinforcement-learning><terminology><math><multi-armed-bandits>
Title: What is a weighted average in a non-stationary k-armed bandit problem?
UserDisplayName: chessprogrammer
ThreadText: # What is a weighted average in a non-stationary k-armed bandit problem?
  
  (Asked by: username_0 on 2018-01-18)
  
  In the book [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/bookdraft2017nov5.pdf) (page 25), by Richard S. Sutton and Andrew G. Barto, there is a discussion of the k-armed bandit problem, where the expected reward from the bandits changes slightly over time (that is, the problem is non-stationary). Instead of updating the Q values by taking an average of all rewards, the book suggests using a constant step-size parameter, so as to give greater weight to more recent rewards. Thus:
  
  $$ Q\_{n+1} = Q\_n + \alpha (R\_n - Q\_n),$$
  
  where $\alpha$ is a constant between 0 and 1.
  
  The book then states that *this a weighted average because the sum of the weights is equal to 1*. What does this mean? Why is this true?
  
  **Tags:** machine-learning, reinforcement-learning, terminology, math, multi-armed-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-01-19. Score: 1 | Accepted answer
  
  The weighted average stands for a linear combination of all values, such that the sum of all weights is 1.
  
  More specifically, if you denote the rewards by a vector $X$, the weighted average will be taking the dot product between $X$ and a vector $W$ such that $0 \le W\_i \le 1$ and the sum of all $W\_i$ is 1.
  
  If each $W\_i = 1/n$ it will be a weighted average (a.k.a the mean). Using the exponential decay $W\_i = \alpha^i/ (\sum W\_i)$ is also a weighted average.
  
  Then both strategies to compute the Q value use weighted average of previous reward.
  
  
  ---
  
  