Answers.Body: <p>what you are talking about are evolutionary strategies, also known as zero order methods</p>
  <p>They try to approximate:
  <span class="math-container">$$
  \nabla_\theta \mathbb{E}^{\pi_\theta}[\sum \gamma^t r_t]
  $$</span>
  and they do so by using a finite difference method, thus perturbing the weights and observing whether the loss increases or not.</p>
  <p>The problem, is that in order to obtain the full gradient <span class="math-container">$\nabla_\theta L$</span> you need <span class="math-container">$|\theta|$</span> perturbations, therefore they poorly scale with todays big models such as neural networks</p>
  <p>However, for certain classes of problems, they work much better than RL (particularly when the loss is very poorly condition), and they are still used</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-08T23:12:52.033Z
Answers.Text: great answer, I was also looking at this the way David suggested. But your point on scalability is excellent. Can you please elaborate on your last paragraph on the loss being very poorly conditioned?
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T00:03:20.250Z
Answers.Text: I’ve just had a thought that aren’t we also making |teta| “perturbations” when we perform backpropagation usually at every training step with neural networks in RL? I’d even go further with the fact the papers I cited suggest applying perturbation only at the beginning of an episode, so I’d say they might result in fewer perturbations than RL algorithms, say TD3. Am I missing something?
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T15:22:50.813Z
Answers.Text: @Mika a the point are the degrees of freedom (span) of the gradient... it's a bit complex to explain it in few words, but think it this way: even if you perturb all the weights, you can either move toward this single perturbation, or away from it, so you have 1 degree of freedom... you will need $|\theta|$ perturbations (possibly orthogonal) to fully characterize the gradient
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T15:24:07.560Z
Answers.Text: @Mika regarding the other one, it's mostly based on personal experience, where if you have a dynamical system (say a robot with many joints), you might have a ill conditioned loss (the loss of the first joint of the robot highly conditions the rest of them)... the same might happen for rewards function with long term consequences
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T16:47:42.357Z
Answers.Text: let me see if I got it right.   In RL back propagation has only one degree of freedom in the form of the loss function and weights are “perturbed” (updated) in the direction of the loss function gradient while in the ES perturbations have much more freedom and directions they can go, right?
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-09T18:43:24.583Z
Answers.Text: @Mika sorry, missunderstood your point… backprop with ppo has obviously all the degrees of freedom, but that gradient is noisy
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T01:19:59.743Z
Answers.Text: can you please refer me to a material with more specific explanation on these methods poorly scaling with large neural networks? I am still a bit confused unless I see the math or more technical explanation. Thanks
Answers.UserDisplayName: Mika
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-11T10:03:34.137Z
Answers.Text: @Mika I'd suggest you to give a look at "Deep RL Bootcamp Lecture 8 Derivative Free Methods" on youtube
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-08T21:22:48.503Z
Answers.LastActivityDate: 2024-07-08T21:22:48.503Z
Answers.UserDisplayName: Alberto
Body: <p>Parameter-based RL methods perturb parameters of a general function approximator directly, rather than adding noise to the resulting actions. Why they haven't received the same attention and gain traction as the SOTA like SAC, TD3, PPO, etc.</p>
  <p>Papers on the parameter-based RL algorithms:</p>
  <ol>
  <li><a href="https://people.idsia.ch/%7Ejuergen/paladyn2010.pdf" rel="nofollow noreferrer">https://people.idsia.ch/~juergen/paladyn2010.pdf</a></li>
  <li><a href="https://mediatum.ub.tum.de/doc/1287490/file.pdf" rel="nofollow noreferrer">https://mediatum.ub.tum.de/doc/1287490/file.pdf</a></li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-08T07:06:41.777Z
Comments.Text: I'd imagine that it is easier intuitively to understand the effects of adding noise to an action than it is to the weights of a deep NN. E.g., adding noise to the weights of a NN could drastically change the action, whereas adding noise to an action is much more interpretable.
Comments.UserDisplayName: David
Tags: <reinforcement-learning><deep-rl>
Title: Why parameter-based RL methods are not widely used?
UserDisplayName: Mika
ThreadText: # Why parameter-based RL methods are not widely used?
  
  (Asked by: username_0 on 2024-07-08)
  
  Parameter-based RL methods perturb parameters of a general function approximator directly, rather than adding noise to the resulting actions. Why they haven't received the same attention and gain traction as the SOTA like SAC, TD3, PPO, etc.
  
  Papers on the parameter-based RL algorithms:
  
  1. [https://people.idsia.ch/~juergen/paladyn2010.pdf](https://people.idsia.ch/%7Ejuergen/paladyn2010.pdf)
  2. <https://mediatum.ub.tum.de/doc/1287490/file.pdf>
  
  **Tags:** reinforcement-learning, deep-rl
  
  > **Comments:**
  > **username_1**: I'd imagine that it is easier intuitively to understand the effects of adding noise to an action than it is to the weights of a deep NN. E.g., adding noise to the weights of a NN could drastically change the action, whereas adding noise to an action is much more interpretable.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-07-08. Score: 2
  
  what you are talking about are evolutionary strategies, also known as zero order methods
  
  They try to approximate:
  $$
  \nabla\_\theta \mathbb{E}^{\pi\_\theta}[\sum \gamma^t r\_t]
  $$
  and they do so by using a finite difference method, thus perturbing the weights and observing whether the loss increases or not.
  
  The problem, is that in order to obtain the full gradient $\nabla\_\theta L$ you need $|\theta|$ perturbations, therefore they poorly scale with todays big models such as neural networks
  
  However, for certain classes of problems, they work much better than RL (particularly when the loss is very poorly condition), and they are still used
  
  > **Comments:**
  > **username_0**: great answer, I was also looking at this the way username_1 suggested. But your point on scalability is excellent. Can you please elaborate on your last paragraph on the loss being very poorly conditioned?  
  > **username_0**: I’ve just had a thought that aren’t we also making |teta| “perturbations” when we perform backpropagation usually at every training step with neural networks in RL? I’d even go further with the fact the papers I cited suggest applying perturbation only at the beginning of an episode, so I’d say they might result in fewer perturbations than RL algorithms, say TD3. Am I missing something?  
  > **username_2**: @username_0 a the point are the degrees of freedom (span) of the gradient... it's a bit complex to explain it in few words, but think it this way: even if you perturb all the weights, you can either move toward this single perturbation, or away from it, so you have 1 degree of freedom... you will need $|\theta|$ perturbations (possibly orthogonal) to fully characterize the gradient  
  > **username_2**: @username_0 regarding the other one, it's mostly based on personal experience, where if you have a dynamical system (say a robot with many joints), you might have a ill conditioned loss (the loss of the first joint of the robot highly conditions the rest of them)... the same might happen for rewards function with long term consequences  
  > **username_0**: let me see if I got it right.   In RL back propagation has only one degree of freedom in the form of the loss function and weights are “perturbed” (updated) in the direction of the loss function gradient while in the ES perturbations have much more freedom and directions they can go, right?  
  > **username_2**: @username_0 sorry, missunderstood your point… backprop with ppo has obviously all the degrees of freedom, but that gradient is noisy  
  > **username_0**: can you please refer me to a material with more specific explanation on these methods poorly scaling with large neural networks? I am still a bit confused unless I see the math or more technical explanation. Thanks  
  > **username_2**: @username_0 I'd suggest you to give a look at "Deep RL Bootcamp Lecture 8 Derivative Free Methods" on youtube  
  
  ---
  
  