Body: <p>I am working on an RL Problem that drives me nuts.
  My goal is to control a robot arm in a simulator that has to do 2 things:</p>
  <ol>
  <li>Hold the arm in a certain position (that is easy and done)</li>
  <li>If I apply an external force, I want the arm to displace a <em>certain amount</em>, which I have defined beforehand as the &quot;stiffness goal&quot;.</li>
  </ol>
  <p>The problems forms a 2 objective optimization problem and I handle that as a sum of weighted rewards, meaning my total reward should be <span class="math-container">$R = w_b*r_b+w_s*r_s$</span> with <span class="math-container">$r$</span> and <span class="math-container">$w$</span> being the rewards and weights for balancing and stiffness.
  However, that function is difficult to give to the agent in practice.</p>
  <p>A reward for the stiffness can only be given at every <code>n</code> timesteps, because in order to do that I have to interact with the system (-&gt; apply external force) and observe the arm's displacement for some other <code>m</code>timesteps. After that I can compute the reward for this displacement with respect to my desired displacement. That is only once per episode in my training setup.
  On the other hand I can easily give the balancing reward at every simulation timestep. That is only based on the current position and that information is always available. But the stiffness reward is only available after I apply external force. So I have a very dense balancing reward, and a very sparse stiffness reward.</p>
  <p>I feel like no matter what I do, the agent just cannot learn the stiffness goal.</p>
  <p>I have tried setting <span class="math-container">$r_s$</span> to 0 at the timesteps where I don't have it, and tried all kinds of different weights if <span class="math-container">$r_s &gt; 0$</span>.
  I have also tried using the last measured stiffness as stiffness for the following timesteps.
  None of this worked.</p>
  <p>Does anybody have any suggestions how to handle such a problem? Right now I am using PPO and an MLP network, with the standard parameters from SB3, except for the learning rate and KL Divergence which I set to <span class="math-container">$10^-5$</span> and <span class="math-container">$10^-4$</span> respectively.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-08T08:27:39.443Z
Comments.Text: Is it possible to reformulate the task to hold the arm in the target position modified by force/stiffness vector? That removes the separate stiffness reward, although it doesn't remove the sparseness of experience (arm still needs to apply ongoing force to maintain position)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-08T10:37:00.060Z
Comments.Text: You mean by sensing the external force, and then adapting to a new target position which is calculated from the desired displacement? I tried that already and it's a bit against the point, also like this the agent does not learn to swing back to it's inital position after the push is over. The balancing target position should be constant the whole time. After the push the agent should "willingly" digress from that target, make the desired swing and go back to the target position.  It's my uni thesis project and definitely poses more problems than i thought.
Comments.UserDisplayName: mavex857
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-08T10:45:52.887Z
Comments.Text: So it's not a force (to balance statically), but an "impulse" (to react dynamically)? How is it being sensed?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-08T12:01:26.183Z
Comments.Text: Yes that's exactly correct. The force is only applied for 1 timestep. It's being sensed by a sensor in the simulator. So I do have the information of when that push happened and with what force.
Comments.UserDisplayName: mavex857
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-08T12:03:32.113Z
Comments.Text: And the force history is kept in the state, available to the agent until no longer required? Otherwise it would be non-Markov. (Or maybe your model is RNN?)
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-08T12:04:42.577Z
Comments.Text: Well I thought about that question too, what exactly my state/observation space is. Right now I am using [position, acceleration, velocity, position target, position error, target stiffness, timesteps from last push, magnitude of last push] as my observation vector and as my state-input for my policy. What exactly do you mean that the history of force has to be in the state?
Comments.UserDisplayName: mavex857
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-08T12:59:06.450Z
Comments.Text: Since I am applying this impulse only once per episode I think the states how I described them are Markov. Also tried using PPO with LSTM network but that didn't seem to work either. Right now I am resorting to hyperparameter optimization but that also does no wonders. I am getting a bit desparate and feel like I've tried about everything...
Comments.UserDisplayName: mavex857
Tags: <reinforcement-learning><optimization><rewards><proximal-policy-optimization><reward-shaping>
Title: Reward shaping for dense and sparse rewards
UserDisplayName: mavex857
ThreadText: # Reward shaping for dense and sparse rewards
  
  (Asked by: username_0 on 2024-03-07)
  
  I am working on an RL Problem that drives me nuts.
  My goal is to control a robot arm in a simulator that has to do 2 things:
  
  1. Hold the arm in a certain position (that is easy and done)
  2. If I apply an external force, I want the arm to displace a *certain amount*, which I have defined beforehand as the "stiffness goal".
  
  The problems forms a 2 objective optimization problem and I handle that as a sum of weighted rewards, meaning my total reward should be $R = w\_b\*r\_b+w\_s\*r\_s$ with $r$ and $w$ being the rewards and weights for balancing and stiffness.
  However, that function is difficult to give to the agent in practice.
  
  A reward for the stiffness can only be given at every `n` timesteps, because in order to do that I have to interact with the system (-> apply external force) and observe the arm's displacement for some other `m`timesteps. After that I can compute the reward for this displacement with respect to my desired displacement. That is only once per episode in my training setup.
  On the other hand I can easily give the balancing reward at every simulation timestep. That is only based on the current position and that information is always available. But the stiffness reward is only available after I apply external force. So I have a very dense balancing reward, and a very sparse stiffness reward.
  
  I feel like no matter what I do, the agent just cannot learn the stiffness goal.
  
  I have tried setting $r\_s$ to 0 at the timesteps where I don't have it, and tried all kinds of different weights if $r\_s > 0$.
  I have also tried using the last measured stiffness as stiffness for the following timesteps.
  None of this worked.
  
  Does anybody have any suggestions how to handle such a problem? Right now I am using PPO and an MLP network, with the standard parameters from SB3, except for the learning rate and KL Divergence which I set to $10^-5$ and $10^-4$ respectively.
  
  **Tags:** reinforcement-learning, optimization, rewards, proximal-policy-optimization, reward-shaping
  
  > **Comments:**
  > **username_1**: Is it possible to reformulate the task to hold the arm in the target position modified by force/stiffness vector? That removes the separate stiffness reward, although it doesn't remove the sparseness of experience (arm still needs to apply ongoing force to maintain position)  
  > **username_0**: You mean by sensing the external force, and then adapting to a new target position which is calculated from the desired displacement? I tried that already and it's a bit against the point, also like this the agent does not learn to swing back to it's inital position after the push is over. The balancing target position should be constant the whole time. After the push the agent should "willingly" digress from that target, make the desired swing and go back to the target position.  It's my uni thesis project and definitely poses more problems than i thought.  
  > **username_1**: So it's not a force (to balance statically), but an "impulse" (to react dynamically)? How is it being sensed?  
  > **username_0**: Yes that's exactly correct. The force is only applied for 1 timestep. It's being sensed by a sensor in the simulator. So I do have the information of when that push happened and with what force.  
  > **username_1**: And the force history is kept in the state, available to the agent until no longer required? Otherwise it would be non-Markov. (Or maybe your model is RNN?)  
  > **username_0**: Well I thought about that question too, what exactly my state/observation space is. Right now I am using [position, acceleration, velocity, position target, position error, target stiffness, timesteps from last push, magnitude of last push] as my observation vector and as my state-input for my policy. What exactly do you mean that the history of force has to be in the state?  
  > **username_0**: Since I am applying this impulse only once per episode I think the states how I described them are Markov. Also tried using PPO with LSTM network but that didn't seem to work either. Right now I am resorting to hyperparameter optimization but that also does no wonders. I am getting a bit desparate and feel like I've tried about everything...  
  
  ---
  
  ## 0 answers
  
  