Answers.Body: <p>RL is not used much in the real industry, as you said because of safety concerns. There are perhaps three different ways of how this would be possible</p>
  <ol>
  <li><p>Use safe exploration to learn a model of the environment and apply model predictive control algorithms on the learned model. This seems to be the approach taken for <a href="https://papers.nips.cc/paper/2018/hash/059fdcd96baeb75112f09fa1dcc740cc-Abstract.html" rel="nofollow noreferrer">data center cooling using model-predictive control</a>.</p>
  </li>
  <li><p>Design an accurate simulation model as a digital twin and use it to train policies there before deploying into the real world. This was the approach taken in the recent breakthrough in nuclear fusion: <a href="https://www.nature.com/articles/s41586-021-04301-9" rel="nofollow noreferrer">Magnetic control of tokamak plasmas through deep reinforcement learning</a></p>
  </li>
  <li><p>Use existing controllers to generate data sets and train RL agents with offline RL (or perhaps imitation learning) and use it as a prior before finetuning it in the real world.</p>
  </li>
  </ol>
  <p>Now, as you mention there is the sim-to-real problem if you train in simulation. The real world usually comes with various uncertainties that can not be simulated well. This may be not extremely relevant for controlled industrial environments but is important in robotics. We can divide the training process into multiple phases, where the agent has less and less information available. The idea is to train a belief model, which aims to assess all the uncertainties and safely adapt to them. An example is:
  <a href="https://arxiv.org/abs/2201.08117" rel="nofollow noreferrer">Learning robust perceptive locomotion for quadrupedal robots in the wild</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-08T18:02:26.443Z
Answers.LastActivityDate: 2022-09-11T01:07:27.603Z
Answers.LastEditDate: 2022-09-11T01:07:27.603Z
Answers.UserDisplayName: matorbi
Body: <p>I'm a newbie to reinforcement learning.
  While studying reinforcement learning, a question arose about how to apply reinforcement learning in the real world.</p>
  <p>Assuming that a reinforcement learning agent trained in simulation is used in the real world, do they usually only use the optimal policy learned in the simulator? Or are they need additional exploration (learning) in the real world?</p>
  <p>In my view, the latter approach is quite impractical in real-world environments where failure by agents' actions is fatal. However, it also seems very difficult for the simulation environment to reflect the real world.</p>
  <p>What approach is being adopted by the industry field?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-08T09:07:32.857Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <reinforcement-learning><deep-rl><training>
Title: How is reinforcement learning applied in the real industry?
UserDisplayName: msms77799
ThreadText: # How is reinforcement learning applied in the real industry?
  
  (Asked by: username_0 on 2022-09-08)
  
  I'm a newbie to reinforcement learning.
  While studying reinforcement learning, a question arose about how to apply reinforcement learning in the real world.
  
  Assuming that a reinforcement learning agent trained in simulation is used in the real world, do they usually only use the optimal policy learned in the simulator? Or are they need additional exploration (learning) in the real world?
  
  In my view, the latter approach is quite impractical in real-world environments where failure by agents' actions is fatal. However, it also seems very difficult for the simulation environment to reflect the real world.
  
  What approach is being adopted by the industry field?
  
  **Tags:** reinforcement-learning, deep-rl, training
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-09-08. Score: 1
  
  RL is not used much in the real industry, as you said because of safety concerns. There are perhaps three different ways of how this would be possible
  
  1. Use safe exploration to learn a model of the environment and apply model predictive control algorithms on the learned model. This seems to be the approach taken for [data center cooling using model-predictive control](https://papers.nips.cc/paper/2018/hash/059fdcd96baeb75112f09fa1dcc740cc-Abstract.html).
  2. Design an accurate simulation model as a digital twin and use it to train policies there before deploying into the real world. This was the approach taken in the recent breakthrough in nuclear fusion: [Magnetic control of tokamak plasmas through deep reinforcement learning](https://www.nature.com/articles/s41586-021-04301-9)
  3. Use existing controllers to generate data sets and train RL agents with offline RL (or perhaps imitation learning) and use it as a prior before finetuning it in the real world.
  
  Now, as you mention there is the sim-to-real problem if you train in simulation. The real world usually comes with various uncertainties that can not be simulated well. This may be not extremely relevant for controlled industrial environments but is important in robotics. We can divide the training process into multiple phases, where the agent has less and less information available. The idea is to train a belief model, which aims to assess all the uncertainties and safely adapt to them. An example is:
  [Learning robust perceptive locomotion for quadrupedal robots in the wild](https://arxiv.org/abs/2201.08117)
  
  
  ---
  
  