Answers.Body: <p>Technical barriers: There should be at least these common sense big barriers:</p>
  <ul>
  <li><strong>Trial-and-error</strong> technique makes the model hard to learn (too many), compared to ready-to-use supervised data</li>
  <li><strong>Number of time-steps</strong> (which usually equals the number of actions of the agent in the trajectory) is large, thus brute-force exploration won't work as the number of trials to find errors is exponential, although negative rewards may help cut short the brute-force tree.</li>
  <li>Real-life RL takes <strong>unlimited number of episodes</strong> (for each episode, a sequence of actions should be learnt), and the incremental training is harder and harder in time with more explored data, unless some past and no-longer-related data are removed, just like humans, we forget some of the past to learn more, remember more the present.</li>
  </ul>
  <p>The technical barriers are at first the barriers to applying them to business. People may produce some supervised data manually rather quick, and thus supervised learning is usually opted first, nobody wish to try RL.</p>
  <p>Harder to find human resources: AI engineers with experiences in supervised learning are more popular and easier to find some; fewer work with RL, thus business projects are not carried out easily if using RL.</p>
  <p>However, from my point of view, RL is very much promising in future as AI entities are now more and more on their own.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T09:56:19.033Z
Answers.LastActivityDate: 2021-01-29T07:04:24.253Z
Answers.LastEditDate: 2021-01-29T07:04:24.253Z
Answers.UserDisplayName: Dan D.
Answers.Body: <p>There is a relatively recent paper that tackles this issue: <a href="https://arxiv.org/pdf/1904.12901.pdf" rel="nofollow noreferrer">Challenges of real-world reinforcement learning</a> (2019) by Gabriel Dulac-Arnold et al., which presents all the challenges that need to be addressed to productionize RL to real world problems, the current approaches/solutions to solve the challenges, and metrics to evaluate them. I will only list them (based on the notes I had taken a few weeks ago). You should read the paper for more details. In any case, for people that are familiar with RL, they will be quite obvious.</p>
  <ol>
  <li>Batch off-line and off-policy training
  <ul>
  <li>One current solution is <em>importance sampling</em></li>
  </ul>
  </li>
  <li>Learning on the real system from limited samples (sample inefficiency)
  <ul>
  <li>Solutions: MAML, use expert demonstrations to bootstrap the agent, model-based approaches</li>
  </ul>
  </li>
  <li>High dimensional continuous state and action spaces
  <ul>
  <li>Solutions: AE-DQN, DRRN</li>
  </ul>
  </li>
  <li><a href="https://arxiv.org/pdf/1606.06565.pdf" rel="nofollow noreferrer">Satisfying safety constraints</a>
  <ul>
  <li>Solutions: constrained MDP, safe exploration strategies, etc.</li>
  </ul>
  </li>
  <li>Partial observability and non-stationarity
  <ul>
  <li>Solutions to partial observability: incorporate history in the observation, recurrent neural networks, etc.</li>
  <li>Solutions to non-stationarity: domain randomization or system identification</li>
  </ul>
  </li>
  <li>Unspecified and multi-objective reward functions
  <ul>
  <li>Solutions: CVaR, Distributional DQN</li>
  </ul>
  </li>
  <li><a href="https://ai.stackexchange.com/q/14224/2444">Explainability</a></li>
  <li>Real-time inference</li>
  <li>System delays (see also <a href="https://ai.stackexchange.com/q/25178/2444">this</a> and <a href="https://ai.stackexchange.com/a/7344/2444">this</a> answers)</li>
  </ol>
  <p>There's also a more recent and related paper <a href="https://arxiv.org/pdf/2003.11881.pdf" rel="nofollow noreferrer">An empirical investigation of the challenges of real-world reinforcement learning</a> (2020) by Gabriel Dulac-Arnold et al, and <a href="https://github.com/google-research/realworldrl_suite" rel="nofollow noreferrer">here</a> you have the associated code with the experiments.</p>
  <p>However, note that RL (in particular, bandits) is already being used to solve at least one real-world problem [<a href="https://netflixtechblog.com/artwork-personalization-c589f074ad76" rel="nofollow noreferrer">1</a>, <a href="https://azure.microsoft.com/en-us/services/cognitive-services/personalizer/" rel="nofollow noreferrer">2</a>]. See also <a href="https://ai.stackexchange.com/a/24355/2444">this</a> answer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-28T11:35:26.447Z
Answers.LastActivityDate: 2021-01-28T16:44:49.480Z
Answers.LastEditDate: 2021-01-28T16:44:49.480Z
Answers.UserDisplayName: nbro
Body: <p>I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, but I don't see the same thing with Reinforcement Learning algorithms.</p>
  <p>What are the biggest barriers to get RL in production?</p>
  
Tags: <reinforcement-learning><deep-rl><applications>
Title: What are the biggest barriers to get RL in production?
UserDisplayName: Alexandre Krul
ThreadText: # What are the biggest barriers to get RL in production?
  
  (Asked by: username_0 on 2021-01-28)
  
  I am studying the state of the art of Reinforcement Learning, and my point is that we see so many applications in the real world using Supervised and Unsupervised learning algorithms in production, but I don't see the same thing with Reinforcement Learning algorithms.
  
  What are the biggest barriers to get RL in production?
  
  **Tags:** reinforcement-learning, deep-rl, applications
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-01-28. Score: 7 | Accepted answer
  
  There is a relatively recent paper that tackles this issue: [Challenges of real-world reinforcement learning](https://arxiv.org/pdf/1904.12901.pdf) (2019) by Gabriel Dulac-Arnold et al., which presents all the challenges that need to be addressed to productionize RL to real world problems, the current approaches/solutions to solve the challenges, and metrics to evaluate them. I will only list them (based on the notes I had taken a few weeks ago). You should read the paper for more details. In any case, for people that are familiar with RL, they will be quite obvious.
  
  1. Batch off-line and off-policy training
     * One current solution is *importance sampling*
  2. Learning on the real system from limited samples (sample inefficiency)
     * Solutions: MAML, use expert demonstrations to bootstrap the agent, model-based approaches
  3. High dimensional continuous state and action spaces
     * Solutions: AE-DQN, DRRN
  4. [Satisfying safety constraints](https://arxiv.org/pdf/1606.06565.pdf)
     * Solutions: constrained MDP, safe exploration strategies, etc.
  5. Partial observability and non-stationarity
     * Solutions to partial observability: incorporate history in the observation, recurrent neural networks, etc.
     * Solutions to non-stationarity: domain randomization or system identification
  6. Unspecified and multi-objective reward functions
     * Solutions: CVaR, Distributional DQN
  7. [Explainability](https://ai.stackexchange.com/q/14224/2444)
  8. Real-time inference
  9. System delays (see also [this](https://ai.stackexchange.com/q/25178/2444) and [this](https://ai.stackexchange.com/a/7344/2444) answers)
  
  There's also a more recent and related paper [An empirical investigation of the challenges of real-world reinforcement learning](https://arxiv.org/pdf/2003.11881.pdf) (2020) by Gabriel Dulac-Arnold et al, and [here](https://github.com/google-research/realworldrl_suite) you have the associated code with the experiments.
  
  However, note that RL (in particular, bandits) is already being used to solve at least one real-world problem [[1](https://netflixtechblog.com/artwork-personalization-c589f074ad76), [2](https://azure.microsoft.com/en-us/services/cognitive-services/personalizer/)]. See also [this](https://ai.stackexchange.com/a/24355/2444) answer.
  
  
  ---
  
  ### Answer by username_1 on 2021-01-28. Score: 4
  
  Technical barriers: There should be at least these common sense big barriers:
  
  * **Trial-and-error** technique makes the model hard to learn (too many), compared to ready-to-use supervised data
  * **Number of time-steps** (which usually equals the number of actions of the agent in the trajectory) is large, thus brute-force exploration won't work as the number of trials to find errors is exponential, although negative rewards may help cut short the brute-force tree.
  * Real-life RL takes **unlimited number of episodes** (for each episode, a sequence of actions should be learnt), and the incremental training is harder and harder in time with more explored data, unless some past and no-longer-related data are removed, just like humans, we forget some of the past to learn more, remember more the present.
  
  The technical barriers are at first the barriers to applying them to business. People may produce some supervised data manually rather quick, and thus supervised learning is usually opted first, nobody wish to try RL.
  
  Harder to find human resources: AI engineers with experiences in supervised learning are more popular and easier to find some; fewer work with RL, thus business projects are not carried out easily if using RL.
  
  However, from my point of view, RL is very much promising in future as AI entities are now more and more on their own.
  
  
  ---
  
  