Answers.Body: <p>Yes, reinforcement learning is very different from supervised learning, the policy (what you call a model) does not know if its predicting right or wrong, or if its taking the correct action or not. In RL there is no concept of "the right action", everything is evaluated through the reward function.</p>
  
  <p>Also there are no ways to compute the ground truth Q-values, if you had that then there is no need to do RL.</p>
  
  <p>In RL you should not think like in supervised learning, there are no error metrics, everything is evaluated on how much accumulated reward the agent receives over an episode.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-19T18:01:14.203Z
Answers.Text: Actually, RL and SL aren't so different or unrelated. In fact, [every SL problem can be cast as an RL one](https://ai.stackexchange.com/a/14168/2444). However, your suggestions are usually valid. Anyway, I think you should address more directly the question "How could I even calculate/know the real Q-value if there is not already a world model existing?" by showing that you don't need to know the true value e.g. in Q-learning.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-19T20:34:32.797Z
Answers.Text: @nbro I did not claim that RL and SL were unrelated, but they are very different. Try doing the opposite, casting a RL problem as a supervised learning one, and you will see the difference. In RL you do not have access to "the right answer".
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-19T20:49:34.377Z
Answers.Text: Yes, I agree with you. I just wanted to point out that they aren't completely unrelated, as one may interpret when reading your first sentence.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-22T11:05:55.947Z
Answers.Text: So, how are the weights updated then? Given a state, when the neural net leads just to a random action, how will the best policy be learned over time?
Answers.UserDisplayName: MScott
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-22T12:33:00.763Z
Answers.Text: @MScott That is a question on its own, there are many techniques to do it (Q-Learning, Policy gradients, etc), a neural network is just used as a function approximation inside those frameworks.
Answers.UserDisplayName: Dr. Snoopy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-19T13:44:41.183Z
Answers.LastActivityDate: 2020-02-19T13:44:41.183Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>How do I calculate the error during the training phase for deep reinforcement learning models?</p>
  
  <p>Deep reinforcement learning is not supervised learning as far as I know. So how can the model know whether it predicts right or wrong? In literature, I find that the "actual" Q-value is calculated, but that sounds like the whole idea behind deep RL is obsolete. How could I even calculate/know the real Q-value if there is not already a world model existing?</p>
  
Tags: <deep-learning><reinforcement-learning><deep-rl>
Title: How to estimate the error during training in deep reinforcement learning
UserDisplayName: MScott
ThreadText: # How to estimate the error during training in deep reinforcement learning
  
  (Asked by: username_0 on 2020-02-19)
  
  How do I calculate the error during the training phase for deep reinforcement learning models?
  
  Deep reinforcement learning is not supervised learning as far as I know. So how can the model know whether it predicts right or wrong? In literature, I find that the "actual" Q-value is calculated, but that sounds like the whole idea behind deep RL is obsolete. How could I even calculate/know the real Q-value if there is not already a world model existing?
  
  **Tags:** deep-learning, reinforcement-learning, deep-rl
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-19. Score: 1
  
  Yes, reinforcement learning is very different from supervised learning, the policy (what you call a model) does not know if its predicting right or wrong, or if its taking the correct action or not. In RL there is no concept of "the right action", everything is evaluated through the reward function.
  
  Also there are no ways to compute the ground truth Q-values, if you had that then there is no need to do RL.
  
  In RL you should not think like in supervised learning, there are no error metrics, everything is evaluated on how much accumulated reward the agent receives over an episode.
  
  > **Comments:**
  > **username_2**: Actually, RL and SL aren't so different or unrelated. In fact, [every SL problem can be cast as an RL one](https://ai.stackexchange.com/a/14168/2444). However, your suggestions are usually valid. Anyway, I think you should address more directly the question "How could I even calculate/know the real Q-value if there is not already a world model existing?" by showing that you don't need to know the true value e.g. in Q-learning.  
  > **username_1**: @username_2 I did not claim that RL and SL were unrelated, but they are very different. Try doing the opposite, casting a RL problem as a supervised learning one, and you will see the difference. In RL you do not have access to "the right answer".  
  > **username_2**: Yes, I agree with you. I just wanted to point out that they aren't completely unrelated, as one may interpret when reading your first sentence.  
  > **username_0**: So, how are the weights updated then? Given a state, when the neural net leads just to a random action, how will the best policy be learned over time?  
  > **username_1**: @username_0 That is a question on its own, there are many techniques to do it (Q-Learning, Policy gradients, etc), a neural network is just used as a function approximation inside those frameworks.  
  
  ---
  
  