Answers.Body: <p>You will see a lot of game examples in reinforcement learning literature, because game environments can often be coded efficiently, and run fast on a single computer that can then contain the environment and the agent. For classic games, such as backgammon, checkers, chess, go, then there are human experts that we can compare results with. Certain games or simplified game-like environments are commonly used to compare different approaches, much like MNIST handwritten digits are used for comparing supervised learning approaches.</p>
  
  <blockquote>
    <p>Is there a way to teach reinforcement learning in applications other than games?</p>
  </blockquote>
  
  <p>Yes. Informally you could apply reinforcement learning approaches whenever you can frame a problem as an agent acting within an environment where it can be informed of the state and a goal-influencing reward value. More formally, reinforcement learning theory is based upon solutions to <a href="https://en.wikipedia.org/wiki/Markov_decision_process" rel="noreferrer">Markov Decision Processes</a>, so if you can fit your problem description to a MDP then the various techniques used in RL - such as Q-learning, SARSA, REINFORCE - can be applied. This fit to theory does not need to be perfect for the resulting system to work, for instance you can often treat unknown or imperfectly observed state as effectively random to the agent, and consider this part of a stochastic environment.</p>
  
  <p>Here are some examples of possible uses for reinforcement learning outside of recreational games:</p>
  
  <ul>
  <li><p>Control logic for motorised robot, such as <a href="https://www.youtube.com/playlist?list=PL5nBAYUyJTrM48dViibyi68urttMlUv7e" rel="noreferrer">learning to flip pancakes and other examples</a>. Here the environment measurements are made by physical sensors on the robot. The rewards are given for completing a goal, but may also be adjusted for smoothness, economic use of energy etc. The agent chooses low-level actions such as motor torque or relay position. In theory there can be nested agents where higher level ones choose the goals for the lower-level ones - e.g. the robot might decide at a high level between doing one of three tasks that require moving to different locations, and at a lower level might be decisions on how to control motors to move the robot to its chosen goal.  </p></li>
  <li><p>Self-driving cars. Although a lot of focus on sensor interpretation - seeing road markings, pedestrians etc, a control system is required in order to select accelerator, brake and steering.</p></li>
  <li><p>Automated financial trading. Perhaps a game to some, there are clear real-world consequences. The reward signal is simple enough though, and RL can be adjusted to prefer long or short term gains.</p></li>
  </ul>
  
  <blockquote>
    <p>is it possible to set this up with say a CAD software?</p>
  </blockquote>
  
  <p>In theory yes, but I do not know what might be available to do this in practice. Also you need one or more goals in mind that you code into the agent (as reward values that it can observe) before giving it a virtual mouse and setting a task to draw something. Computer games come with a reward scheme built in as their scoring system, and provide frequent feedback, so an agent can gain knowledge of good vs bad decisions quickly. You would need to replace this scoring component with something that represents your goals for the CAD-based system. </p>
  
  <p>CAD does not have anything suitable built-in, although CAD tools with simulations, such as various physics engines or finite element analysis, could allow you to score designs based on a simulated physical measure. Other possibilities include analysis of strain, non-wasteful use of material, whatever metrics the CAD/CAM system can provide for a partial or completed design. The tricky part is constraining a design to its goal or purpose and either arranging for that to be rewarded, or building the constraints into the environment; giving an RL agent full unconstrained control of CAD process and rewarding on lowest strain will likely result in something very uninteresting such as a small cube.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-07-30T16:07:49.537Z
Answers.LastActivityDate: 2018-06-18T05:50:32.677Z
Answers.LastEditDate: 2018-06-18T05:50:32.677Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Reinforcement learning (and, in particular, bandit) algorithms have been and can be used to solve problems other than games, such as</p>
  <ul>
  <li><a href="http://rob.schapire.net/papers/www10.pdf" rel="nofollow noreferrer">Recommender systems</a> (actually used in practice by e.g. <a href="https://netflixtechblog.com/artwork-personalization-c589f074ad76" rel="nofollow noreferrer">Netflix</a> or <a href="https://azure.microsoft.com/en-us/services/cognitive-services/personalizer/" rel="nofollow noreferrer">Microsoft</a>)</li>
  <li><a href="https://arxiv.org/pdf/1909.09571.pdf" rel="nofollow noreferrer">Portfolio optimization</a></li>
  <li><a href="https://www.pnas.org/content/pnas/106/52/22387.full.pdf" rel="nofollow noreferrer">Clinical trials</a></li>
  <li><a href="https://arxiv.org/pdf/1611.01578.pdf" rel="nofollow noreferrer">Hyper-parameter optimization</a></li>
  <li>Self-driving cars (although I am not aware of any real self-driving car that uses just reinforcement learning; however, in principle, RL can be used in this context too)</li>
  </ul>
  <p>In general, any problem that can be modelled as the maximization of some notion of <em>reward</em>, where you need to interact with some environment (with some states) by taking some actions, can, <em>in principle</em>, be solved by reinforcement learning. Take a look at <a href="https://arxiv.org/pdf/1908.06973.pdf" rel="nofollow noreferrer">this pre-print paper</a> (2019) for other applications.</p>
  <p>However, note that there are several obstacles that prevent RL algorithms from being widely adopted to solve real-world problems, starting from poor sample complexity (i.e. they require many samples to reach a good performance) or the partial inability to evaluate their performance online without affecting the users.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-01T14:39:27.063Z
Answers.Text: Your answer doesn't reflect reality when it comes to systems in the real world. Even the papers you link are based on experiments with recorded data not real time systems. I have worked with professors in RL and even they say it's pretty much a disaster in real systems which is why the robotics community largely rejects it even though you see its cherry picked performance in youtube videos.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-01T14:41:00.690Z
Answers.Text: @FourierFlux Bandits are used by Netflix, so RL is used in the real-world too. Check this https://netflixtechblog.com/artwork-personalization-c589f074ad76. However, note that I am not saying that they are extensively used. You're right that there are many obstacles that prevent RL from being more widely used.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-01T14:42:46.347Z
Answers.Text: Bandit problems aren't really what people think about when talking about RL. It's a very simplified case.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-01T14:43:09.477Z
Answers.Text: @FourierFlux Well, Bandit problems are just special cases of the full RL problem, so they are RL problems too.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-01T01:20:24.500Z
Answers.LastActivityDate: 2020-11-01T14:50:45.430Z
Answers.LastEditDate: 2020-11-01T14:50:45.430Z
Answers.UserDisplayName: nbro
Body: <p>Is there a way to teach reinforcement learning in applications other than games? </p>
  
  <p>The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the games via the reinforcement network. Is it possible to set this up with say a CAD software?</p>
  
Tags: <reinforcement-learning><applications>
Title: Are there any applications of reinforcement learning other than games?
UserDisplayName: Mark Markrowave Charlton
ThreadText: # Are there any applications of reinforcement learning other than games?
  
  (Asked by: username_0 on 2017-06-16)
  
  Is there a way to teach reinforcement learning in applications other than games?
  
  The only examples I can find on the Internet are of game agents. I understand that VNC's control the input to the games via the reinforcement network. Is it possible to set this up with say a CAD software?
  
  **Tags:** reinforcement-learning, applications
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2017-07-30. Score: 6
  
  You will see a lot of game examples in reinforcement learning literature, because game environments can often be coded efficiently, and run fast on a single computer that can then contain the environment and the agent. For classic games, such as backgammon, checkers, chess, go, then there are human experts that we can compare results with. Certain games or simplified game-like environments are commonly used to compare different approaches, much like MNIST handwritten digits are used for comparing supervised learning approaches.
  
  > Is there a way to teach reinforcement learning in applications other than games?
  
  Yes. Informally you could apply reinforcement learning approaches whenever you can frame a problem as an agent acting within an environment where it can be informed of the state and a goal-influencing reward value. More formally, reinforcement learning theory is based upon solutions to [Markov Decision Processes](https://en.wikipedia.org/wiki/Markov_decision_process), so if you can fit your problem description to a MDP then the various techniques used in RL - such as Q-learning, SARSA, REINFORCE - can be applied. This fit to theory does not need to be perfect for the resulting system to work, for instance you can often treat unknown or imperfectly observed state as effectively random to the agent, and consider this part of a stochastic environment.
  
  Here are some examples of possible uses for reinforcement learning outside of recreational games:
  
  * Control logic for motorised robot, such as [learning to flip pancakes and other examples](https://www.youtube.com/playlist?list=PL5nBAYUyJTrM48dViibyi68urttMlUv7e). Here the environment measurements are made by physical sensors on the robot. The rewards are given for completing a goal, but may also be adjusted for smoothness, economic use of energy etc. The agent chooses low-level actions such as motor torque or relay position. In theory there can be nested agents where higher level ones choose the goals for the lower-level ones - e.g. the robot might decide at a high level between doing one of three tasks that require moving to different locations, and at a lower level might be decisions on how to control motors to move the robot to its chosen goal.
  * Self-driving cars. Although a lot of focus on sensor interpretation - seeing road markings, pedestrians etc, a control system is required in order to select accelerator, brake and steering.
  * Automated financial trading. Perhaps a game to some, there are clear real-world consequences. The reward signal is simple enough though, and RL can be adjusted to prefer long or short term gains.
  
  > is it possible to set this up with say a CAD software?
  
  In theory yes, but I do not know what might be available to do this in practice. Also you need one or more goals in mind that you code into the agent (as reward values that it can observe) before giving it a virtual mouse and setting a task to draw something. Computer games come with a reward scheme built in as their scoring system, and provide frequent feedback, so an agent can gain knowledge of good vs bad decisions quickly. You would need to replace this scoring component with something that represents your goals for the CAD-based system.
  
  CAD does not have anything suitable built-in, although CAD tools with simulations, such as various physics engines or finite element analysis, could allow you to score designs based on a simulated physical measure. Other possibilities include analysis of strain, non-wasteful use of material, whatever metrics the CAD/CAM system can provide for a partial or completed design. The tricky part is constraining a design to its goal or purpose and either arranging for that to be rewarded, or building the constraints into the environment; giving an RL agent full unconstrained control of CAD process and rewarding on lowest strain will likely result in something very uninteresting such as a small cube.
  
  
  ---
  
  ### Answer by username_2 on 2020-11-01. Score: 3
  
  Reinforcement learning (and, in particular, bandit) algorithms have been and can be used to solve problems other than games, such as
  
  * [Recommender systems](http://rob.schapire.net/papers/www10.pdf) (actually used in practice by e.g. [Netflix](https://netflixtechblog.com/artwork-personalization-c589f074ad76) or [Microsoft](https://azure.microsoft.com/en-us/services/cognitive-services/personalizer/))
  * [Portfolio optimization](https://arxiv.org/pdf/1909.09571.pdf)
  * [Clinical trials](https://www.pnas.org/content/pnas/106/52/22387.full.pdf)
  * [Hyper-parameter optimization](https://arxiv.org/pdf/1611.01578.pdf)
  * Self-driving cars (although I am not aware of any real self-driving car that uses just reinforcement learning; however, in principle, RL can be used in this context too)
  
  In general, any problem that can be modelled as the maximization of some notion of *reward*, where you need to interact with some environment (with some states) by taking some actions, can, *in principle*, be solved by reinforcement learning. Take a look at [this pre-print paper](https://arxiv.org/pdf/1908.06973.pdf) (2019) for other applications.
  
  However, note that there are several obstacles that prevent RL algorithms from being widely adopted to solve real-world problems, starting from poor sample complexity (i.e. they require many samples to reach a good performance) or the partial inability to evaluate their performance online without affecting the users.
  
  > **Comments:**
  > **username_3**: Your answer doesn't reflect reality when it comes to systems in the real world. Even the papers you link are based on experiments with recorded data not real time systems. I have worked with professors in RL and even they say it's pretty much a disaster in real systems which is why the robotics community largely rejects it even though you see its cherry picked performance in youtube videos.  
  > **username_2**: @FourierFlux Bandits are used by Netflix, so RL is used in the real-world too. Check this https://netflixtechblog.com/artwork-personalization-c589f074ad76. However, note that I am not saying that they are extensively used. You're right that there are many obstacles that prevent RL from being more widely used.  
  > **username_3**: Bandit problems aren't really what people think about when talking about RL. It's a very simplified case.  
  > **username_2**: @FourierFlux Well, Bandit problems are just special cases of the full RL problem, so they are RL problems too.  
  
  ---
  
  