Answers.Body: <p>In reinforcement learning, to learn off policy control, you need data on the states, actions and rewards at each time step. If, in addition to a recorded video, you had a recording of controller inputs, and could add reward data by hand, then you could use a standard reinforcement learning method, e.g. DQN. Simply run the DQN training loop as normal, but skip the parts where it acts in the environment, and only train on batches of recorded experience.</p>
  
  <p>With <em>only</em> video data, your options are limited. However, it might still be useful, because a significant part of the challenge is a machine vision task. For a DQN agent, it will need to convert frames from the video (e.g. last 4 frames) into a prediction of the different rewards that it could get depending on which controller buttons are pressed. If you can teach a separate neural network to perform a vision task on relevant video data, it may may help. You could use the learned weights from the first layers of this network as the starting point for your Q values network, and it will likely speed up a DQN figuring out the relationship to its predictions. This sort of task switch following learning is called <a href="https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a" rel="nofollow noreferrer">transfer learning</a>, and is often used in computer vision tasks.</p>
  
  <p>A possibly useful starting task if you have a video, but no controller or reward data, would be to predict the next frame(s) of the video, given say four starting frames (you need more than one so that the neural network can use velocity information). It should be possible to generate the training data using opencv or ffmpeg from your recordings.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-19T14:07:17.107Z
Answers.Text: So first I train a network to identify important objects? (Can I use tho YOLO network here?) And with the trained identifier I train a second network to predict the next frame from four predecessor frames.
  
  But then? And will this really give me an advantage? I can still let the agent play for itselfs. I think I will have enougth work setting up the enviroment and rewards and so on...
Answers.UserDisplayName: Voß
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-19T14:13:19.047Z
Answers.Text: @bruh: If you have information about the objects available that might be useful as a transfer learning starter and you could use e.g. YOLO. However, my suggestion in the answer is simply to predict the next frame, because you already have that in the video data, so it requires a lot less data preparation. Yes you can start directly with the RL, that is what the DQN authors did for all the Atari games, and it worked well for most of them. However, you asked whether it was possible to learn from just the video, and this answer addresses the original question.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-19T09:51:13.987Z
Answers.LastActivityDate: 2019-12-19T10:26:51.957Z
Answers.LastEditDate: 2019-12-19T10:26:51.957Z
Answers.UserDisplayName: Neil Slater
Body: <p>My goal is to train an agent to play MarioKart on the Nintendo DS. My first approach (in theory) was to setup an emulator on my pc and let the agent play for ages. But then a colleague suggested to train the agent first on pre recorded humanly played video data, to achieve some sort of base level. And then for further perfection let the agent play for its own with the emulator.</p>
  <p>But I have no clue how training with video data works. E.g. I wonder how to calculate a loss since there is no reward. Or am I getting the intuition wrong?</p>
  <p>I would appreciate it if someone could explain this technique to me.</p>
  
Tags: <reinforcement-learning>
Title: How does reinforcement learning with video data work?
UserDisplayName: Voß
ThreadText: # How does reinforcement learning with video data work?
  
  (Asked by: username_0 on 2019-12-19)
  
  My goal is to train an agent to play MarioKart on the Nintendo DS. My first approach (in theory) was to setup an emulator on my pc and let the agent play for ages. But then a colleague suggested to train the agent first on pre recorded humanly played video data, to achieve some sort of base level. And then for further perfection let the agent play for its own with the emulator.
  
  But I have no clue how training with video data works. E.g. I wonder how to calculate a loss since there is no reward. Or am I getting the intuition wrong?
  
  I would appreciate it if someone could explain this technique to me.
  
  **Tags:** reinforcement-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-19. Score: 2 | Accepted answer
  
  In reinforcement learning, to learn off policy control, you need data on the states, actions and rewards at each time step. If, in addition to a recorded video, you had a recording of controller inputs, and could add reward data by hand, then you could use a standard reinforcement learning method, e.g. DQN. Simply run the DQN training loop as normal, but skip the parts where it acts in the environment, and only train on batches of recorded experience.
  
  With *only* video data, your options are limited. However, it might still be useful, because a significant part of the challenge is a machine vision task. For a DQN agent, it will need to convert frames from the video (e.g. last 4 frames) into a prediction of the different rewards that it could get depending on which controller buttons are pressed. If you can teach a separate neural network to perform a vision task on relevant video data, it may may help. You could use the learned weights from the first layers of this network as the starting point for your Q values network, and it will likely speed up a DQN figuring out the relationship to its predictions. This sort of task switch following learning is called [transfer learning](https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a), and is often used in computer vision tasks.
  
  A possibly useful starting task if you have a video, but no controller or reward data, would be to predict the next frame(s) of the video, given say four starting frames (you need more than one so that the neural network can use velocity information). It should be possible to generate the training data using opencv or ffmpeg from your recordings.
  
  > **Comments:**
  > **username_0**: So first I train a network to identify important objects? (Can I use tho YOLO network here?) And with the trained identifier I train a second network to predict the next frame from four predecessor frames.  But then? And will this really give me an advantage? I can still let the agent play for itselfs. I think I will have enougth work setting up the enviroment and rewards and so on...  
  > **username_1**: @bruh: If you have information about the objects available that might be useful as a transfer learning starter and you could use e.g. YOLO. However, my suggestion in the answer is simply to predict the next frame, because you already have that in the video data, so it requires a lot less data preparation. Yes you can start directly with the RL, that is what the DQN authors did for all the Atari games, and it worked well for most of them. However, you asked whether it was possible to learn from just the video, and this answer addresses the original question.  
  
  ---
  
  