Answers.Body: <p>PPO is an on-policy algorithm so you <strong>must</strong> use trajectories generated by the current policy.</p>
  <p>DQN is an off-policy algorithm, so you could add these trajectories to the buffer, but you also need &quot;bad&quot; trajectories so that you try to cover the state-action space. Of course you can't cover the full space, but you need both good and bad examples.</p>
  <p>If you have a way of generating &quot;good&quot; trajectories, then maybe try some imitiation learning algorithms like behaviour cloning or DAgger.</p>
  <p>You could also try using imitation learning only for pre-training, and once your policy is good enough to generate &quot;reasonable&quot; trajectories, you can train further with PPO.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T17:10:07.123Z
Answers.Text: How would you continue training with PPO afterwards? If you setup a DQN to do the imitation learning part then how do you copy those parameters into the PPO network. I think I am able to generate a good amount of pretty good trajectories for my problem. What would be the difference between using those in a DQN network versus using the imitation learning you have suggested?
Answers.UserDisplayName: Erik Storm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T20:41:48.533Z
Answers.Text: Imitation learning is different from DQN. In DQN you train a Q-network using the replay buffer. With IL you will train a policy network using supervised learning (see Behavior cloning or DAgger). Once you finish training with IL you can start training with PPO. The PPO algorithm does not require you to start with a random policy. You can start with any policy. In particular you will start PPO with the policy that was trained with IL.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-05T10:38:57.840Z
Answers.LastActivityDate: 2023-09-26T19:09:27.743Z
Answers.LastEditDate: 2023-09-26T19:09:27.743Z
Answers.UserDisplayName: pi-tau
Body: <p>I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that are satisfactory and are found while using no neural network. Is it possible that I then use those trajectories in the replay buffer of a DQN or PPO and then update the neural network a couple of times? This would initialise the neural network in kinda the right direction and would be a bit like imitation learning. Does someone know if this is a viable idea or if there is another way of doing this?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-04T14:59:15.910Z
Comments.Text: PPO no, DQN yes (though there is a caveat that DQN might still not work, particularly if you’re stopping all online interactions with the environment)
Comments.UserDisplayName: David
Tags: <reinforcement-learning><dqn><pytorch><proximal-policy-optimization>
Title: Filling replay buffer with expert trajectories for PPO/DQN
UserDisplayName: Erik Storm
ThreadText: # Filling replay buffer with expert trajectories for PPO/DQN
  
  (Asked by: username_0 on 2023-09-04)
  
  I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that are satisfactory and are found while using no neural network. Is it possible that I then use those trajectories in the replay buffer of a DQN or PPO and then update the neural network a couple of times? This would initialise the neural network in kinda the right direction and would be a bit like imitation learning. Does someone know if this is a viable idea or if there is another way of doing this?
  
  **Tags:** reinforcement-learning, dqn, pytorch, proximal-policy-optimization
  > **Comments:**
  > **username_1**: PPO no, DQN yes (though there is a caveat that DQN might still not work, particularly if you’re stopping all online interactions with the environment)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-09-05. Score: 2
  
  PPO is an on-policy algorithm so you **must** use trajectories generated by the current policy.
  
  DQN is an off-policy algorithm, so you could add these trajectories to the buffer, but you also need "bad" trajectories so that you try to cover the state-action space. Of course you can't cover the full space, but you need both good and bad examples.
  
  If you have a way of generating "good" trajectories, then maybe try some imitiation learning algorithms like behaviour cloning or DAgger.
  
  You could also try using imitation learning only for pre-training, and once your policy is good enough to generate "reasonable" trajectories, you can train further with PPO.
  
  > **Comments:**
  > **username_0**: How would you continue training with PPO afterwards? If you setup a DQN to do the imitation learning part then how do you copy those parameters into the PPO network. I think I am able to generate a good amount of pretty good trajectories for my problem. What would be the difference between using those in a DQN network versus using the imitation learning you have suggested?  
  > **username_2**: Imitation learning is different from DQN. In DQN you train a Q-network using the replay buffer. With IL you will train a policy network using supervised learning (see Behavior cloning or DAgger). Once you finish training with IL you can start training with PPO. The PPO algorithm does not require you to start with a random policy. You can start with any policy. In particular you will start PPO with the policy that was trained with IL.  
  
  ---
  
  