Answers.Body: <p>Short answer: <strong>Yes, it is.</strong></p>
  
  <p><strong>Explanation</strong></p>
  
  <p>Reinforcement learning can be considered as a online learning. That is, you can train your model with a single data/reward pairs.  As with any online learning algorithm, there are a few things to consider.</p>
  
  <p>The model tends to forget the knowledge gained. To overcome this problem, one can save new data in a circular buffer called history and train the model with a portion of mix of new and old data. This is actually the common way to train an RL model and can be adopted to real-time systems. There are also others <a href="https://medium.com/datadriveninvestor/online-learning-adapting-to-a-changing-environment-without-catastrophically-forgetting-the-past-fdf2f7157391" rel="nofollow noreferrer">techniques</a> to overcome it.</p>
  
  <p>Another problem is that if only one data point is fed to the network, it will be impossible to apply some techniques, such as Batch normalization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T19:43:40.687Z
Answers.Text: This doesn't address the massive, massive number of simulations required. IIRC the number of walking simulations for robots is way, way higher than the real time constraints imposed on animals learning to walk.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T19:54:49.273Z
Answers.Text: Where does this requirement come from? You can pre-train your model and then apply online learning so that it is always up to date.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T20:22:34.053Z
Answers.Text: The issue is IMO, real systems have built in priors I think(or models which better lead to correct parameter estimation) and it seems this is missing from the RL paradigm. Like the convergence rate for animal/human behavior is astronomically faster than any type of RL policy.
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T20:35:23.103Z
Answers.Text: All this simulation are required to collect data. Theoretically, your model have to visit all states of the world and learn reward for each action in given state. If we train a model using Q-learning algorithm, it doesn't matter how you train your model. You have a bunch of `(input, action, reward)` tuples. You may feed this tuple as long as you get new one. In this case a network just an approximation of a [Q-function](https://en.wikipedia.org/wiki/Q-learning#Algorithm)
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T20:45:29.923Z
Answers.Text: But do you dispute the fact that humans don't need nearly the same number of trials to reach a viable policy?
Answers.UserDisplayName: FourierFlux
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T20:56:42.177Z
Answers.Text: We don't know how our brain works. Q-learning has Markov assumption. It means that current state doesn't depend on previous states, which means that the model doesn't have memory.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-01T12:37:34.980Z
Answers.Text: Humans may not need as much online training as we are capable of transferring knowledge from different domains to help in achieving the task
Answers.UserDisplayName: KaneM
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T12:35:24.317Z
Answers.LastActivityDate: 2020-02-29T12:35:24.317Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>From what I have seen, any results involving RL almost always take a massive number of simulations to reach a remotely good policy.</p>
  
  <p>Will any form of RL be viable for real-time systems? </p>
  
Tags: <reinforcement-learning><applications><real-time>
Title: Is reinforcement learning suited for real-time systems?
UserDisplayName: FourierFlux
ThreadText: # Is reinforcement learning suited for real-time systems?
  
  (Asked by: username_0 on 2020-02-29)
  
  From what I have seen, any results involving RL almost always take a massive number of simulations to reach a remotely good policy.
  
  Will any form of RL be viable for real-time systems?
  
  **Tags:** reinforcement-learning, applications, real-time
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-29. Score: 1
  
  Short answer: **Yes, it is.**
  
  **Explanation**
  
  Reinforcement learning can be considered as a online learning. That is, you can train your model with a single data/reward pairs. As with any online learning algorithm, there are a few things to consider.
  
  The model tends to forget the knowledge gained. To overcome this problem, one can save new data in a circular buffer called history and train the model with a portion of mix of new and old data. This is actually the common way to train an RL model and can be adopted to real-time systems. There are also others [techniques](https://medium.com/datadriveninvestor/online-learning-adapting-to-a-changing-environment-without-catastrophically-forgetting-the-past-fdf2f7157391) to overcome it.
  
  Another problem is that if only one data point is fed to the network, it will be impossible to apply some techniques, such as Batch normalization.
  
  > **Comments:**
  > **username_0**: This doesn't address the massive, massive number of simulations required. IIRC the number of walking simulations for robots is way, way higher than the real time constraints imposed on animals learning to walk.  
  > **username_1**: Where does this requirement come from? You can pre-train your model and then apply online learning so that it is always up to date.  
  > **username_0**: The issue is IMO, real systems have built in priors I think(or models which better lead to correct parameter estimation) and it seems this is missing from the RL paradigm. Like the convergence rate for animal/human behavior is astronomically faster than any type of RL policy.  
  > **username_1**: All this simulation are required to collect data. Theoretically, your model have to visit all states of the world and learn reward for each action in given state. If we train a model using Q-learning algorithm, it doesn't matter how you train your model. You have a bunch of `(input, action, reward)` tuples. You may feed this tuple as long as you get new one. In this case a network just an approximation of a [Q-function](https://en.wikipedia.org/wiki/Q-learning#Algorithm)  
  > **username_0**: But do you dispute the fact that humans don't need nearly the same number of trials to reach a viable policy?  
  > **username_1**: We don't know how our brain works. Q-learning has Markov assumption. It means that current state doesn't depend on previous states, which means that the model doesn't have memory.  
  > **username_2**: Humans may not need as much online training as we are capable of transferring knowledge from different domains to help in achieving the task  
  
  ---
  
  