Answers.Body: <p>The critical detail of the initial multi-k-armed bandit as it is described in the quoted text, is that the agent cannot detect or use knowledge of <em>which</em> bandit problem it is presented with on each time step.</p>
  <p>Therefore the expected reward from action <span class="math-container">$a$</span> is the weighted sum of expected rewards from all bandits for that action. It is weighted according to the random selection, and stays fixed for the combined bandit, same as for the individual bandits that it is composed of. This is a single real value for expected reward, which means optimising it looks identical from the outside to optimising a simpler bandit with less &quot;machinery&quot;.</p>
  <p>All of this changes as soon as the agent is allowed to observe data, i.e. the context of which sub-distribution it is taking an action for in the next time step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-12T19:35:20.183Z
Answers.Text: This answer is not clear for me as well. I would be happy if it can be updated.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-12T19:36:33.023Z
Answers.Text: @DSPinfinity What exactly is not clear? It is pretty thorough IMO. I probably could not do better without prompting what the problem is. You may need someone else to answer
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-13T07:07:17.400Z
Answers.Text: I think it will be clear if wording is changed and more explanations are added.
Answers.UserDisplayName: DSPinfinity
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-01T13:29:53.047Z
Answers.LastActivityDate: 2022-05-01T13:29:53.047Z
Answers.UserDisplayName: Neil Slater
Body: <p>Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):</p>
  <blockquote>
  <p>As an example, suppose there are several different k-armed bandit tasks, and that on
  each step you confront one of these chosen at random. Thus, the bandit task changes
  randomly from step to step. If the probabilities with which each task is selected for you
  do not change over time, this would appear as a single stationary k-armed bandit task,
  and you could use one of the methods described in this chapter.</p>
  </blockquote>
  <p>If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?</p>
  
Tags: <reinforcement-learning><sutton-barto><multi-armed-bandits>
Title: If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?
UserDisplayName: user3489173
ThreadText: # If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?
  
  (Asked by: username_0 on 2022-04-30)
  
  Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):
  
  > As an example, suppose there are several different k-armed bandit tasks, and that on
  > each step you confront one of these chosen at random. Thus, the bandit task changes
  > randomly from step to step. If the probabilities with which each task is selected for you
  > do not change over time, this would appear as a single stationary k-armed bandit task,
  > and you could use one of the methods described in this chapter.
  
  If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?
  
  **Tags:** reinforcement-learning, sutton-barto, multi-armed-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-01. Score: 1
  
  The critical detail of the initial multi-k-armed bandit as it is described in the quoted text, is that the agent cannot detect or use knowledge of *which* bandit problem it is presented with on each time step.
  
  Therefore the expected reward from action $a$ is the weighted sum of expected rewards from all bandits for that action. It is weighted according to the random selection, and stays fixed for the combined bandit, same as for the individual bandits that it is composed of. This is a single real value for expected reward, which means optimising it looks identical from the outside to optimising a simpler bandit with less "machinery".
  
  All of this changes as soon as the agent is allowed to observe data, i.e. the context of which sub-distribution it is taking an action for in the next time step.
  
  > **Comments:**
  > **username_2**: This answer is not clear for me as well. I would be happy if it can be updated.  
  > **username_1**: @DSPinfinity What exactly is not clear? It is pretty thorough IMO. I probably could not do better without prompting what the problem is. You may need someone else to answer  
  > **username_2**: I think it will be clear if wording is changed and more explanations are added.  
  
  ---
  
  