Answers.Body: <p>Let's have a look at the introduction of <em>Chapter 2: Multi-armed Bandits</em> in the
  <a href="http://incompleteideas.net/book/bookdraft2017nov5.pdf" rel="nofollow noreferrer">Reinforcement Learning: An Introduction by Sutton, Barto</a></p>
  <blockquote>
  <p><em><strong>The most important feature distinguishing reinforcement learning from other types of learning is that it uses training information that evaluates the actions taken rather than instructs by giving correct
  actions.</strong> This is what creates the need for active exploration, for an explicit search for good behavior.
  <strong>Purely evaluative feedback indicates how good the action taken was, but not whether it was the best or
  the worst action possible. Purely instructive feedback, on the other hand, indicates the correct action
  to take, independently of the action actually taken.</strong> This kind of feedback is the basis of supervised
  learning, which includes large parts of pattern classification, artificial neural networks, and system
  identification. In their pure forms, these two kinds of feedback are quite distinct: evaluative feedback
  depends entirely on the action taken, whereas instructive feedback is independent of the action taken.
  In this chapter <strong>we study the evaluative aspect of reinforcement learning in a simplified setting</strong>, one
  that does not involve learning to act in more than one situation. This nonassociative setting is the
  one in which most prior work involving evaluative feedback has been done, and it avoids much of the
  complexity of the full reinforcement learning problem. Studying this case enables us to see most clearly
  how evaluative feedback differs from, and yet can be combined with, instructive feedback.
  The particular nonassociative, <strong>evaluative feedback problem that we explore is a simple version of
  the k-armed bandit problem</strong>. We use this problem to introduce a number of basic learning methods
  which we extend in later chapters to apply to the <strong>full reinforcement learning problem</strong>. At the end
  of this chapter, we take a step closer to the full reinforcement learning problem by discussing what
  happens when the bandit problem becomes associative, that is, when actions are taken in more than
  one situation.</em></p>
  </blockquote>
  <p>Since bandits involve <em>evaluative feedback</em> they are indeed a type of a (simplified) reinforcement learning problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T20:16:29.353Z
Answers.Text: Thanks @Tomasz Bartkowiak. I accepted nbro’s answer, as it is more comprehensive, in the sense that it includes views from authors primarily working in the full RL setup (the ones you already referenced) and authors primarily working on bandits.
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T15:06:37.607Z
Answers.LastActivityDate: 2020-05-03T15:06:37.607Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Tomasz Bartkowiak
Answers.Body: <p>Several important researchers distinguish between bandit problems and the general reinforcement learning problem.</p>
  
  <p>The book <a href="http://incompleteideas.net/book/RLbook2020.pdf#page=45" rel="noreferrer">Reinforcement learning: an introduction</a> by Sutton and Barto describes bandit problems as a special case of the general RL problem.</p>
  
  <blockquote>
    <p>The first chapter of this part of the book describes solution methods for the <strong>special case of the reinforcement learning problem</strong> in which there is <strong>only a single state</strong>, called <strong>bandit problems</strong>. The second chapter describes the <strong>general problem formulation</strong> that we treat throughout the rest of the book — <strong>finite Markov decision processes</strong> — and its main ideas including Bellman equations and value functions.</p>
  </blockquote>
  
  <p>This means that you can represent your bandit problem as an MDP with a single state and possibly multiple actions.</p>
  
  <p>In <a href="https://tor-lattimore.com/downloads/book/book.pdf#page=21" rel="noreferrer">section 1.1.2 of the book Bandit Algorithms</a> (2020), Szepesvari and Lattimore describe the differences between bandits and reinforcement learning</p>
  
  <blockquote>
    <p>One of the distinguishing features of all bandit problems studied in this book is that <strong>the learner never needs to plan for the future</strong>. More precisely, we will invariably make the assumption that the learner's available choices and rewards tomorrow are not affected by their decisions today. <strong>Problems that do require this kind of long-term planning fall into the realm of reinforcement learning</strong></p>
  </blockquote>
  
  <p>This definition is different than the one by Sutton and Barto. In this case, only bandit problems where the learner doesn't need to plan for the future are considered.</p>
  
  <p>In any case, bandit problems and RL problems have a lot of similarities. For example, both attempt to deal with the exploration-exploitation trade-off and, in both cases, the underlying problem can be formulated as a Markov decision process.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-03T20:15:38.447Z
Answers.Text: Thanks @nbro . In fact, in one of his lectures on bandits, Tor Lattimore referred to bandits as “baby RL”, but I wouldn’t use that term in a formal context :) Szepesvari also includes bandit algorithms in his book Algorithms for Reinforcement Learning (https://sites.ualberta.ca/~szepesva/rlbook.html , Section 4.2.1). In any case, I think the key difference is as they mention it “the learner's available choices and rewards tomorrow are not affected by their decisions today.”
Answers.UserDisplayName: user5093249
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-24T13:10:07.867Z
Answers.Text: If there is only one state in the bandit problem, so what is it actually? Is it possible to describe it in detail? Give a cartpole problem, the state could be cart position, cart velocity, pole angle, pole angular velocity. In the bandit problem, my understanding is that the state can be the combination of the pattern, if you get a specific pattern you get the corresponding reward. However, it is hard to model and not only one. So when we say there is only one state, what we are referring to?
Answers.UserDisplayName: Ynjxsjmh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-02T22:55:58.180Z
Answers.LastActivityDate: 2020-05-03T00:28:24.240Z
Answers.LastEditDate: 2020-05-03T00:28:24.240Z
Answers.UserDisplayName: nbro
Body: <p>If a research paper uses multi-armed bandits (either in their standard or contextual form) to solve a particular task, can we say that they solved this task using a reinforcement learning approach? Or should we distinguish between the two and use the RL term only when it is associated with an MDP formulation?</p>
  
  <p>In fact, each RL course/textbook usually contains a section about bandits (especially when dealing with the exploration-exploitation tradeoff). Additionally, bandits also have the concept of actions and rewards. </p>
  
  <p>I just want to make sure what the right terminology should be, when describing either approach.</p>
  
Tags: <reinforcement-learning><terminology><multi-armed-bandits><contextual-bandits>
Title: Are bandits considered an RL approach?
UserDisplayName: user5093249
ThreadText: # Are bandits considered an RL approach?
  
  (Asked by: username_0 on 2020-05-02)
  
  If a research paper uses multi-armed bandits (either in their standard or contextual form) to solve a particular task, can we say that they solved this task using a reinforcement learning approach? Or should we distinguish between the two and use the RL term only when it is associated with an MDP formulation?
  
  In fact, each RL course/textbook usually contains a section about bandits (especially when dealing with the exploration-exploitation tradeoff). Additionally, bandits also have the concept of actions and rewards.
  
  I just want to make sure what the right terminology should be, when describing either approach.
  
  **Tags:** reinforcement-learning, terminology, multi-armed-bandits, contextual-bandits
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-05-02. Score: 7 | Accepted answer
  
  Several important researchers distinguish between bandit problems and the general reinforcement learning problem.
  
  The book [Reinforcement learning: an introduction](http://incompleteideas.net/book/RLbook2020.pdf#page=45) by Sutton and Barto describes bandit problems as a special case of the general RL problem.
  
  > The first chapter of this part of the book describes solution methods for the **special case of the reinforcement learning problem** in which there is **only a single state**, called **bandit problems**. The second chapter describes the **general problem formulation** that we treat throughout the rest of the book — **finite Markov decision processes** — and its main ideas including Bellman equations and value functions.
  
  This means that you can represent your bandit problem as an MDP with a single state and possibly multiple actions.
  
  In [section 1.1.2 of the book Bandit Algorithms](https://tor-lattimore.com/downloads/book/book.pdf#page=21) (2020), Szepesvari and Lattimore describe the differences between bandits and reinforcement learning
  
  > One of the distinguishing features of all bandit problems studied in this book is that **the learner never needs to plan for the future**. More precisely, we will invariably make the assumption that the learner's available choices and rewards tomorrow are not affected by their decisions today. **Problems that do require this kind of long-term planning fall into the realm of reinforcement learning**
  
  This definition is different than the one by Sutton and Barto. In this case, only bandit problems where the learner doesn't need to plan for the future are considered.
  
  In any case, bandit problems and RL problems have a lot of similarities. For example, both attempt to deal with the exploration-exploitation trade-off and, in both cases, the underlying problem can be formulated as a Markov decision process.
  
  > **Comments:**
  > **username_0**: Thanks @username_2 . In fact, in one of his lectures on bandits, Tor Lattimore referred to bandits as “baby RL”, but I wouldn’t use that term in a formal context :) Szepesvari also includes bandit algorithms in his book Algorithms for Reinforcement Learning (https://sites.ualberta.ca/~szepesva/rlbook.html , Section 4.2.1). In any case, I think the key difference is as they mention it “the learner's available choices and rewards tomorrow are not affected by their decisions today.”  
  > **username_3**: If there is only one state in the bandit problem, so what is it actually? Is it possible to describe it in detail? Give a cartpole problem, the state could be cart position, cart velocity, pole angle, pole angular velocity. In the bandit problem, my understanding is that the state can be the combination of the pattern, if you get a specific pattern you get the corresponding reward. However, it is hard to model and not only one. So when we say there is only one state, what we are referring to?  
  
  ---
  
  ### Answer by username_1 on 2020-05-03. Score: 3
  
  Let's have a look at the introduction of *Chapter 2: Multi-armed Bandits* in the
  [Reinforcement Learning: An Introduction by Sutton, Barto](http://incompleteideas.net/book/bookdraft2017nov5.pdf)
  
  > ***The most important feature distinguishing reinforcement learning from other types of learning is that it uses training information that evaluates the actions taken rather than instructs by giving correct
  > actions.** This is what creates the need for active exploration, for an explicit search for good behavior.
  > **Purely evaluative feedback indicates how good the action taken was, but not whether it was the best or
  > the worst action possible. Purely instructive feedback, on the other hand, indicates the correct action
  > to take, independently of the action actually taken.** This kind of feedback is the basis of supervised
  > learning, which includes large parts of pattern classification, artificial neural networks, and system
  > identification. In their pure forms, these two kinds of feedback are quite distinct: evaluative feedback
  > depends entirely on the action taken, whereas instructive feedback is independent of the action taken.
  > In this chapter **we study the evaluative aspect of reinforcement learning in a simplified setting**, one
  > that does not involve learning to act in more than one situation. This nonassociative setting is the
  > one in which most prior work involving evaluative feedback has been done, and it avoids much of the
  > complexity of the full reinforcement learning problem. Studying this case enables us to see most clearly
  > how evaluative feedback differs from, and yet can be combined with, instructive feedback.
  > The particular nonassociative, **evaluative feedback problem that we explore is a simple version of
  > the k-armed bandit problem**. We use this problem to introduce a number of basic learning methods
  > which we extend in later chapters to apply to the **full reinforcement learning problem**. At the end
  > of this chapter, we take a step closer to the full reinforcement learning problem by discussing what
  > happens when the bandit problem becomes associative, that is, when actions are taken in more than
  > one situation.*
  
  Since bandits involve *evaluative feedback* they are indeed a type of a (simplified) reinforcement learning problem.
  
  > **Comments:**
  > **username_0**: Thanks @username_1. I accepted username_2’s answer, as it is more comprehensive, in the sense that it includes views from authors primarily working in the full RL setup (the ones you already referenced) and authors primarily working on bandits.  
  
  ---
  
  