Answers.Body: <p>In theory on-policy algorithms like PPO and actor-critic should perform better than off-policy algorithms such as DQN. The downside is that you will need to collect new data once the policy is updated, while with DQN you have a replay buffer that reuses experiences.</p>
  <p><a href="https://pi-tau.github.io/posts/actor-critic/" rel="nofollow noreferrer">PPO</a> also allows you to make several updates before collecting new data thus increasing the sample efficiency of policy gradient methods. But still DQN is way more efficient. So if your environment is slow, not parallelizable, or you cant even simulate it and have to use the real thing, then maybe use DQN.</p>
  <p>Another major difference between DQN and direct policy optimization methdos (like PPO) is that you cannot use DQN with continuous action spaces. There is just no way to compute the maximum over the action space in the equation <span class="math-container">$Q(s_t,a_t) = r_{t+1} + \gamma \max_{a \in \mathcal{A}}Q(s_{t+1},a)$</span>. Policy gradient methods have no problem with this.</p>
  <p>Regarding exploration, there are a lot of great works that research this topic both for policy gradient methods and for DQN-style off-policy methods. I can't really say which line of research has better results.</p>
  <p>Finally, regarding your problem, I wanted to say that if you implement the algorithms yourself there might be a great chance that you implemented something incorrectly. Especially in the case of PPO, which is known to have a lot of details not mentioned in the official paper. Using a correct implementation is very important for hard problems. You might be able to solve atari pendulum, but it will fail on harder tasks. <a href="https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html#tips-and-tricks-when-implementing-an-rl-algorithm" rel="nofollow noreferrer">Here</a> is a nice suggestion for testing your implementation on increasingly harder environments: CartPole, LunarLander, Pong, Breakout.</p>
  <p>You can also check out these two blogposts and compare your implementation of PPO:</p>
  <p><a href="https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/" rel="nofollow noreferrer">https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/</a></p>
  <p><a href="https://pi-tau.github.io/posts/actor-critic/" rel="nofollow noreferrer">https://pi-tau.github.io/posts/actor-critic/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T16:31:18.137Z
Answers.LastActivityDate: 2023-09-12T16:38:25.787Z
Answers.LastEditDate: 2023-09-12T16:38:25.787Z
Answers.UserDisplayName: pi-tau
Body: <p>I have tried many different RL architectures: DQN, PPO, Policy optimization, and for my specific problem they all failed in their basic setup. Eventually I discovered that my problem had too sparse/deceptive rewards and that they would never work. This made me wonder what the practical differences are between the architectures. In my view it looks like that the more recent architectures have better exploration/exploitation trade offs. And that the main problem with RL is not finding any good trajectories to learn from, but that all algorithms are pretty good at performing once it has found that good trajectory. So is the exploration/exploitation trade off the biggest difference on which most architectures differ on?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-12T15:55:31.757Z
Comments.Text: those are algorithms, not architectures, and they differ in the ability to learn... however, if your reward signal is a +1/-1 after 382739857 actions, it's veeery hard for all of them to learn, as there is no "supervision" signal to optimize for... you might want to look at keywords like "reward shaping" and "intrinsic curiosity"
Comments.UserDisplayName: Alberto
Tags: <reinforcement-learning><architecture><exploration-exploitation-tradeoff>
Title: Practical differences between the different RL architectures
UserDisplayName: Erik Storm
ThreadText: # Practical differences between the different RL architectures
  
  (Asked by: username_0 on 2023-09-12)
  
  I have tried many different RL architectures: DQN, PPO, Policy optimization, and for my specific problem they all failed in their basic setup. Eventually I discovered that my problem had too sparse/deceptive rewards and that they would never work. This made me wonder what the practical differences are between the architectures. In my view it looks like that the more recent architectures have better exploration/exploitation trade offs. And that the main problem with RL is not finding any good trajectories to learn from, but that all algorithms are pretty good at performing once it has found that good trajectory. So is the exploration/exploitation trade off the biggest difference on which most architectures differ on?
  
  **Tags:** reinforcement-learning, architecture, exploration-exploitation-tradeoff
  > **Comments:**
  > **username_1**: those are algorithms, not architectures, and they differ in the ability to learn... however, if your reward signal is a +1/-1 after 382739857 actions, it's veeery hard for all of them to learn, as there is no "supervision" signal to optimize for... you might want to look at keywords like "reward shaping" and "intrinsic curiosity"  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-09-12. Score: 1
  
  In theory on-policy algorithms like PPO and actor-critic should perform better than off-policy algorithms such as DQN. The downside is that you will need to collect new data once the policy is updated, while with DQN you have a replay buffer that reuses experiences.
  
  [PPO](https://pi-tau.github.io/posts/actor-critic/) also allows you to make several updates before collecting new data thus increasing the sample efficiency of policy gradient methods. But still DQN is way more efficient. So if your environment is slow, not parallelizable, or you cant even simulate it and have to use the real thing, then maybe use DQN.
  
  Another major difference between DQN and direct policy optimization methdos (like PPO) is that you cannot use DQN with continuous action spaces. There is just no way to compute the maximum over the action space in the equation $Q(s\_t,a\_t) = r\_{t+1} + \gamma \max\_{a \in \mathcal{A}}Q(s\_{t+1},a)$. Policy gradient methods have no problem with this.
  
  Regarding exploration, there are a lot of great works that research this topic both for policy gradient methods and for DQN-style off-policy methods. I can't really say which line of research has better results.
  
  Finally, regarding your problem, I wanted to say that if you implement the algorithms yourself there might be a great chance that you implemented something incorrectly. Especially in the case of PPO, which is known to have a lot of details not mentioned in the official paper. Using a correct implementation is very important for hard problems. You might be able to solve atari pendulum, but it will fail on harder tasks. [Here](https://stable-baselines.readthedocs.io/en/master/guide/rl_tips.html#tips-and-tricks-when-implementing-an-rl-algorithm) is a nice suggestion for testing your implementation on increasingly harder environments: CartPole, LunarLander, Pong, Breakout.
  
  You can also check out these two blogposts and compare your implementation of PPO:
  
  <https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/>
  
  <https://pi-tau.github.io/posts/actor-critic/>
  
  
  ---
  
  