Answers.Body: <p>MNIST (along with CIFAR) may be the &quot;Hello World&quot; of <strong>supervised</strong> learning for image classification, but it is definitely not the &quot;Hello World&quot; of all machine learning techniques, given that RL is also part of ML and MNIST is definitely not the &quot;Hello World&quot; of RL.</p>
  <p>I don't think there is a single &quot;Hello World&quot; problem for RL. However, if you are looking for simple problems (or environments) that are usually used as baselines to assess the quality of RL agents, then I would say that the simple <strong>grid worlds</strong> where you need to move from one place to the other, the <strong>CartPole</strong>, <strong>MountainCar</strong>, <strong>Pendulum</strong> or other environments listed <a href="https://github.com/openai/gym/wiki/Leaderboard" rel="noreferrer">here</a> are often used.</p>
  <p>The environment that you choose to train and test your RL agent depends on your goals. For example, if you designed an algorithm that is supposed to deal with continuous action spaces, then an environment where you can take only a discrete number of actions may not be a good option.</p>
  <p>The mentioned environments are very simple (i.e. toy problems). In my opinion, we need more serious environments that can show the applicability of RL to other areas other than (relatively simple) games.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-13T13:28:29.507Z
Answers.LastActivityDate: 2020-09-13T20:26:37.147Z
Answers.LastEditDate: 2020-09-13T20:26:37.147Z
Answers.UserDisplayName: nbro
Answers.Body: <p>While there's no simple Hello World problem of RL, if your aim is to understand the basic working of Reinforcement Learning and see it at play while using as few moving parts as possible, a simple suggestion would be using <strong>Tabular</strong> Q-Learning in a toy environment (like your suggested Cart-Pole Env).</p>
  <p><strong>Here's the reasoning behind this suggestion</strong></p>
  <p>Let's say we interpret MNIST's label as a <em>Hello World</em> of Supervised Learning to mean something showing the basic steps of doing Supervised Learning: Create a model, load the data, then train.</p>
  <p>If that interpretation is not far off, we can say a simple introductory problem to Reinforcement Learning (RL) should focus on easily demonstrating a working <a href="https://www.coursera.org/lecture/fundamentals-of-reinforcement-learning/markov-decision-processes-8T0GQ" rel="nofollow noreferrer">Markov Decision Process (MDP)</a> which is the backbone of the RL decision making process. As such, this minimal working would involve: Observing the world, selecting an action, as shown in this loop:</p>
  <p><a href="https://i.sstatic.net/rrgVV.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rrgVV.jpg" alt="Simple MDP" /></a></p>
  <p>This picture is missing two important steps in an RL algorithm learning loop:</p>
  <ol>
  <li>Estimating the rewards or Fitting the model</li>
  <li>Improving how you select actions. (Updating your policy)</li>
  </ol>
  <p>How we decide to update the policy, or fit the model is what makes difference in the RL algorithm most of the time.</p>
  <p>So a suggested first problem would be one that helps you <strong>see the MDP in action</strong>, while keeping steps 1 and 2 simple enough so that you <strong>understand how the agent learns</strong>. Tabular Q-Learning seems clear enough for this because it uses a <a href="https://observablehq.com/@frynd/q-table-reinforcement-learning" rel="nofollow noreferrer">Q-table</a> represented as a 2D array to do the two steps. This should not suggest Q-learning is a &quot;Hello World&quot; RL algorithm because of the said relative ease in understanding it :)</p>
  <p>You will be unable to use it's Tabular version anywhere else than in a toy environment though, typically Frozen-Lake and CartPole. An improvement would be <a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" rel="nofollow noreferrer">using a neural network</a> instead of a table to estimate Q values.</p>
  <p>Here are a few useful resources:</p>
  <ol>
  <li><a href="https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0" rel="nofollow noreferrer">Q-Learning with Tables</a> (Guide)</li>
  <li><a href="https://gist.github.com/awjuliani/9024166ca08c489a60994e529484f7fe#file-q-table-learning-clean-ipynb" rel="nofollow noreferrer">Q-learning jupyter notebook</a> (Code ~25 lines)</li>
  <li><a href="https://github.com/mugoh/reinforcementLearning/tree/master/algorithms/qlearning" rel="nofollow noreferrer">Q-Learning with Frozen-Lake and Taxi</a> (Code)</li>
  <li><a href="https://www.freecodecamp.org/news/diving-deeper-into-reinforcement-learning-with-q-learning-c18d0db58efe/" rel="nofollow noreferrer">Reinforcement Learning with Q-Learning</a> (Guide)</li>
  </ol>
  <p>A multi-armed bandit would also be great in introducing you to <a href="https://or.stackexchange.com/questions/4598/difference-between-exploration-and-exploitation-in-simulated-annealing-algorithm">exploration-exploitation trade-off</a> (which Q-learning does too), though it wouldn't be considered a full RL algorithm since it has no context.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-14T13:40:14.277Z
Answers.Text: So, the ML version of `while(!worldOnFire){flames++}`?
Answers.UserDisplayName: Mast
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-14T22:09:50.703Z
Answers.Text: Although you are talking about **tabular** Q-learning, I think that saying that Q-learning is the "Hello World" **program** (not **problem**, as in my answer) of RL is misleading, because it makes it seem like Q-learning is just a "toy useless algorithm", but, of course, it's not. Q-learning **with function approximation** (specifically, neural networks) was used to solve/play the Atari games with superhuman performance in many cases. So, Q-learning, although quite simple, can be used to solve non-trivial problems, so it's a powerful algorithm, and that's why your suggestion is misleading.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-15T06:43:26.830Z
Answers.Text: Definitely, if this answer presents Q-learning as a "useless toy algorithm", that's very misleading. I've noted how the use of program and problem (as you explain it) contributes to this. I've attempted to edit the answer to alleviate that, as much as I could. Thanks @nbro! My intention was not to mislead.
Answers.UserDisplayName: mugoh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-14T08:56:56.680Z
Answers.LastActivityDate: 2020-09-15T06:32:51.710Z
Answers.LastEditDate: 2020-09-15T06:32:51.710Z
Answers.UserDisplayName: mugoh
Body: <p>As we all know, <a href="https://en.wikipedia.org/wiki/%22Hello,_World!%22_program" rel="noreferrer">&quot;Hello World&quot;</a> is usually the first program that any programmer learns/implements in any language/framework.</p>
  <p>As Aurélien Géron mentioned in his <a href="https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/" rel="noreferrer">book</a> that <a href="http://yann.lecun.com/exdb/mnist/" rel="noreferrer">MNIST</a> is often called the <em>Hello World of Machine Learning</em>, <strong>is there any &quot;Hello World&quot; problem of Reinforcement Learning?</strong></p>
  <p>A few candidates that I could think of are <a href="https://en.wikipedia.org/wiki/Multi-armed_bandit" rel="noreferrer">multi armed bandits problem</a> and <a href="https://gym.openai.com/envs/CartPole-v0/" rel="noreferrer">Cart Pole Env</a>.</p>
  
Tags: <machine-learning><reinforcement-learning>
Title: What is the "Hello World" problem of Reinforcement Learning?
UserDisplayName: Arpit-Gole
ThreadText: # What is the "Hello World" problem of Reinforcement Learning?
  
  (Asked by: username_0 on 2020-09-13)
  
  As we all know, ["Hello World"](https://en.wikipedia.org/wiki/%22Hello,_World!%22_program) is usually the first program that any programmer learns/implements in any language/framework.
  
  As Aurélien Géron mentioned in his [book](https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/) that [MNIST](http://yann.lecun.com/exdb/mnist/) is often called the *Hello World of Machine Learning*, **is there any "Hello World" problem of Reinforcement Learning?**
  
  A few candidates that I could think of are [multi armed bandits problem](https://en.wikipedia.org/wiki/Multi-armed_bandit) and [Cart Pole Env](https://gym.openai.com/envs/CartPole-v0/).
  
  **Tags:** machine-learning, reinforcement-learning
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-09-13. Score: 10
  
  MNIST (along with CIFAR) may be the "Hello World" of **supervised** learning for image classification, but it is definitely not the "Hello World" of all machine learning techniques, given that RL is also part of ML and MNIST is definitely not the "Hello World" of RL.
  
  I don't think there is a single "Hello World" problem for RL. However, if you are looking for simple problems (or environments) that are usually used as baselines to assess the quality of RL agents, then I would say that the simple **grid worlds** where you need to move from one place to the other, the **CartPole**, **MountainCar**, **Pendulum** or other environments listed [here](https://github.com/openai/gym/wiki/Leaderboard) are often used.
  
  The environment that you choose to train and test your RL agent depends on your goals. For example, if you designed an algorithm that is supposed to deal with continuous action spaces, then an environment where you can take only a discrete number of actions may not be a good option.
  
  The mentioned environments are very simple (i.e. toy problems). In my opinion, we need more serious environments that can show the applicability of RL to other areas other than (relatively simple) games.
  
  
  ---
  
  ### Answer by username_2 on 2020-09-14. Score: 2
  
  While there's no simple Hello World problem of RL, if your aim is to understand the basic working of Reinforcement Learning and see it at play while using as few moving parts as possible, a simple suggestion would be using **Tabular** Q-Learning in a toy environment (like your suggested Cart-Pole Env).
  
  **Here's the reasoning behind this suggestion**
  
  Let's say we interpret MNIST's label as a *Hello World* of Supervised Learning to mean something showing the basic steps of doing Supervised Learning: Create a model, load the data, then train.
  
  If that interpretation is not far off, we can say a simple introductory problem to Reinforcement Learning (RL) should focus on easily demonstrating a working [Markov Decision Process (MDP)](https://www.coursera.org/lecture/fundamentals-of-reinforcement-learning/markov-decision-processes-8T0GQ) which is the backbone of the RL decision making process. As such, this minimal working would involve: Observing the world, selecting an action, as shown in this loop:
  
  [![Simple MDP](https://i.sstatic.net/rrgVV.jpg)](https://i.sstatic.net/rrgVV.jpg)
  
  This picture is missing two important steps in an RL algorithm learning loop:
  
  1. Estimating the rewards or Fitting the model
  2. Improving how you select actions. (Updating your policy)
  
  How we decide to update the policy, or fit the model is what makes difference in the RL algorithm most of the time.
  
  So a suggested first problem would be one that helps you **see the MDP in action**, while keeping steps 1 and 2 simple enough so that you **understand how the agent learns**. Tabular Q-Learning seems clear enough for this because it uses a [Q-table](https://observablehq.com/@frynd/q-table-reinforcement-learning) represented as a 2D array to do the two steps. This should not suggest Q-learning is a "Hello World" RL algorithm because of the said relative ease in understanding it :)
  
  You will be unable to use it's Tabular version anywhere else than in a toy environment though, typically Frozen-Lake and CartPole. An improvement would be [using a neural network](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html) instead of a table to estimate Q values.
  
  Here are a few useful resources:
  
  1. [Q-Learning with Tables](https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0) (Guide)
  2. [Q-learning jupyter notebook](https://gist.github.com/awjuliani/9024166ca08c489a60994e529484f7fe#file-q-table-learning-clean-ipynb) (Code ~25 lines)
  3. [Q-Learning with Frozen-Lake and Taxi](https://github.com/mugoh/reinforcementLearning/tree/master/algorithms/qlearning) (Code)
  4. [Reinforcement Learning with Q-Learning](https://www.freecodecamp.org/news/diving-deeper-into-reinforcement-learning-with-q-learning-c18d0db58efe/) (Guide)
  
  A multi-armed bandit would also be great in introducing you to [exploration-exploitation trade-off](https://or.stackexchange.com/questions/4598/difference-between-exploration-and-exploitation-in-simulated-annealing-algorithm) (which Q-learning does too), though it wouldn't be considered a full RL algorithm since it has no context.
  
  > **Comments:**
  > **username_3**: So, the ML version of `while(!worldOnFire){flames++}`?  
  > **username_1**: Although you are talking about **tabular** Q-learning, I think that saying that Q-learning is the "Hello World" **program** (not **problem**, as in my answer) of RL is misleading, because it makes it seem like Q-learning is just a "toy useless algorithm", but, of course, it's not. Q-learning **with function approximation** (specifically, neural networks) was used to solve/play the Atari games with superhuman performance in many cases. So, Q-learning, although quite simple, can be used to solve non-trivial problems, so it's a powerful algorithm, and that's why your suggestion is misleading.  
  > **username_2**: Definitely, if this answer presents Q-learning as a "useless toy algorithm", that's very misleading. I've noted how the use of program and problem (as you explain it) contributes to this. I've attempted to edit the answer to alleviate that, as much as I could. Thanks @nbro! My intention was not to mislead.  
  
  ---
  
  