Body: <p>DQN for Atari takes considerable training time. For example, the 2015 <a href="https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf" rel="nofollow noreferrer">paper</a> in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total. One reason is that DQN for image data typically uses a CNN, which is costly to train.</p>
  <p>However, the main purpose of a CNN is to extract the image features. Note that the policy for DQN is represented by a CNN and an output layer equal to the number of discrete actions. Is it possible to use a pretrained DQN to accelerate the training process by fixing the weights of the underlying pretrained CNN, resetting the weights of the output layer, and then running another (possibly different) DQN algorithm to relearn the weights of the output layer? Both DQN algorithms would be run on the same underlying environment.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-29T15:58:29.283Z
Comments.Text: Could you please cite or explain why DQN for Atari takes 7+ days for training? Is it because you are working on a certain Atari environment or using a specific DQN architecture?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-29T15:59:22.850Z
Comments.Text: Natures DQN. I use Dopamine”s implementation. What is your experience for Atari?
Comments.UserDisplayName: YoYO Man
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-29T16:06:00.347Z
Comments.Text: What do you intend to pre-train DQN on? Another environment?
Comments.UserDisplayName: DeepQZero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-29T16:08:29.033Z
Comments.Text: Same environment. I am trying to use my own implementation of RL algorithm, by removing the last layer of a pretrained DQN. (The preceding layers are just CNN that does the image feature extraction)
Comments.UserDisplayName: YoYO Man
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-01T12:49:31.400Z
Comments.Text: For a Bachelor's project, a friend and I did the same; and getting satisfactory results never took far more than 7 days or so. To get really good scores, yes. But the onset of training could be observed much earlier. 
  But if you just want to pretrain a CNN, what you could do, is, for example, developing an autoencoder that you train on re-creating some atari-game frames. I guess you are working on the Gym environment? Then, sampling some training data for an autoencoder should not be a problem. Then, you could just take the encoder of the autoencoder as your CNN.
Comments.UserDisplayName: Daniel B.
Tags: <reinforcement-learning><convolutional-neural-networks><dqn>
Title: Atari Games: Pretrained CNN to accelerate training?
UserDisplayName: YoYO Man
ThreadText: # Atari Games: Pretrained CNN to accelerate training?
  
  (Asked by: username_0 on 2020-06-29)
  
  DQN for Atari takes considerable training time. For example, the 2015 [paper](https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf) in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total. One reason is that DQN for image data typically uses a CNN, which is costly to train.
  
  However, the main purpose of a CNN is to extract the image features. Note that the policy for DQN is represented by a CNN and an output layer equal to the number of discrete actions. Is it possible to use a pretrained DQN to accelerate the training process by fixing the weights of the underlying pretrained CNN, resetting the weights of the output layer, and then running another (possibly different) DQN algorithm to relearn the weights of the output layer? Both DQN algorithms would be run on the same underlying environment.
  
  **Tags:** reinforcement-learning, convolutional-neural-networks, dqn
  > **Comments:**
  > **username_1**: Could you please cite or explain why DQN for Atari takes 7+ days for training? Is it because you are working on a certain Atari environment or using a specific DQN architecture?  
  > **username_0**: Natures DQN. I use Dopamine”s implementation. What is your experience for Atari?  
  > **username_1**: What do you intend to pre-train DQN on? Another environment?  
  > **username_0**: Same environment. I am trying to use my own implementation of RL algorithm, by removing the last layer of a pretrained DQN. (The preceding layers are just CNN that does the image feature extraction)  
  > **username_2**: For a Bachelor's project, a friend and I did the same; and getting satisfactory results never took far more than 7 days or so. To get really good scores, yes. But the onset of training could be observed much earlier.  But if you just want to pretrain a CNN, what you could do, is, for example, developing an autoencoder that you train on re-creating some atari-game frames. I guess you are working on the Gym environment? Then, sampling some training data for an autoencoder should not be a problem. Then, you could just take the encoder of the autoencoder as your CNN.  
  
  ---
  
  ## 0 answers
  
  