Answers.Body: <p>I do not have enough reputation to just comment, but here are my thoughts:</p>
  <p>Do you actually need to use subgoals/waypoints?
  If your objective is simply how to train an agent to be able to accomplish such a task, then I am thinking you do not need to break it into subgoals/waypoints.
  Indeed, the environment you describe can be reduced to a 2d environment with a starting position and a single goal position, where the agent must find the shortest path between these two positions. Since the barrel must be lifted first whatever happens, this can be hard coded in the reset point (I don't think there is much use in training a RL agent to learn to just lift an object as there are not many ways this can be done!), same for the drop at the goal position (simply set the routine for the drop once the agent has reached the goal position).</p>
  <p>If you actually need waypoints, then as @Luca Anzalone suggested in the comments, you can still include the goal's coordinate in your states.</p>
  <p>For Domain Randomization, if the agent does not observe the positions (and size?) of the obstacles, then the policy will not be able to know where it can move (this is due to the overly-cautious or &quot;conservative&quot; nature of DR policies). The only way this would work is if your obstacles are distributed such that there is one specific path that is always opened. Otherwise, you should consider adding the obstacle information to your states (the agent does not need to observe the full space, but it should be able to at least observe when it is close to an obstacle, so that it can avoid it or move away).</p>
  <p>If you really need to train the policy for lifting and dropping, then you could take a look at continual reinforcement learning (specifically Multi-task Learning?), but I think that even in this case, a properly designed reward function could be enough (one that penalizes the agent for starting to move while the barrel is still on the ground [of course this means a 3d environment instead of the 2d case I mentioned above] - the penalty can be proportional to the height of the barrel and maybe to the distance from the starting or goal position).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-25T09:26:41.603Z
Answers.Text: Many thanks to your interesting answer/comment. But the problem is that, as you pointed out, the reward function will be a sum of many different factors, which in worst case work against each other. So the barrel could move perfectly on the vertical but getting a penalty because it moves far away from the final goal (e.g. the goal is at the same level as the start position)
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-26T03:20:39.013Z
Answers.Text: @Dave Yes, reward shaping is well known problem in RL... There isn't many ways to solve the issue. Have you considered mutli-task reinforcement learning? Another possibility for training a single policy is multi-objective reinforcement learning in which the task reward and penalty are weighted by some preference vector. In particular, take a look at dynamics weight MORL algorithms: [link](https://proceedings.mlr.press/v97/abels19a). The paper is for Q-learning, but you can adapt it to SAC or PPO by simply using universal policy networks.
Answers.UserDisplayName: Ahnel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-27T07:38:24.413Z
Answers.Text: Your intuition is correct man. But I still struggle in understanding the concept. That a scalar reward too less for complex robot system is, is clear. I read the paper carefully and looking for some simple tutorial on its implementation. But yeah, that's probably the way to go
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-25T07:55:54.387Z
Answers.LastActivityDate: 2024-07-25T07:55:54.387Z
Answers.UserDisplayName: Ahnel
Body: <p>To be able to explain my question I thought it is probably better to consider the following example: Let's take an environment, where a bridge crane need to lift a barrel from the position &quot;start&quot; and move it to the position &quot;goal&quot; moving along the axes: <span class="math-container">$X, Y, Z$</span>.</p>
  <p><a href="https://i.sstatic.net/YvdRVMx7.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YvdRVMx7.jpg" alt="example" /></a></p>
  <p>But the movement cannot be straight to the goal. The movement from the start to the goal is based on the presence of obstable in the enviroment and some pratical rules (e.g. the barrel must be lifted first, then horizontally moved and at the end dropped at the goal position).</p>
  <p>I'm struggling to figure out, how to train an agent to be able to accomplish such a task.
  But here I want to share my thoughts:</p>
  <ol>
  <li>At first I thought to use PPO or SAC as model free algorithms and assigning different <em>rewards</em>, when reached the goal and the positions in between (let's call them <em>waypoints</em>) In other words, the agent starts from the <em>start</em> position and gets a reward of +100 points if the agent reaches <span class="math-container">$a$</span> in some time. Now... in order to move forward I would need to give a higher reward to <span class="math-container">$b$</span>. But that means, that the agent will try during the training to reach <span class="math-container">$b$</span> directly instead going to <span class="math-container">$b$</span> over <span class="math-container">$a$</span>.</li>
  <li>Even if the issues in the first point are somehow solvable there is the problem, that the agent would learn to reach the goal through the waypoints seen during the training, even if the new environment is now different (obstacles are in different position, the waypoints changed their positions, etc.). So the agent would learn &quot;memorizing&quot; the training environment, without generalization.</li>
  </ol>
  <p>What could be the best strategy in such enviroments?
  Thanks</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-19T16:07:14.973Z
Comments.Text: Have you tried to combine PPO/SAC (or similar) with *Hindsight Experience Replay*? HER allows to train goal-based policies. Maybe you can split the problem in subgoals, having also a goal-dependent reward function. For the generalization issue you may need to apply *domain randomization* and/or extend the state representation to include information about the obstacles in the environment. Also,I think that randomizing the start/goal positions as well the obstacles during training should help
Comments.UserDisplayName: Luca Anzalone
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-20T08:50:49.877Z
Comments.Text: There are many things to unpack here:
  1) PPO is onpolicy HER is for offpolicy algorihtms. So it could fit at best with SAC
  2) Yes I tried SAC + HER: it miserably fails. The reason is that in case of HER I need a sparse reward (e.g. +1 if the (sub) goal is reached, 0 otherwise). Since the parameter space is continuous there are too many states to be explored, it simply does not work. For this reason I switched back to shaping the reward instead using a sparsed one
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-20T08:53:08.353Z
Comments.Text: 3) You say. "split the problem in subgoals". But this is actually the point. How? Should I train the agent to move first up. And when it does it, it should learns to move horizontally?
  4) Domain randomization is what I'm doing now. The result is that the agent does not converge to a solution
Comments.UserDisplayName: Dave
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-22T07:27:50.630Z
Comments.Text: Maybe you can try to train multiple policies (agents), and then coordinate them by hand-made rules: you can have an agent that lifts the barrel up, then another specialized for moving it horizontally, and eventually another than puts the barrel on the ground. Maybe this can help. Also including the goal's coordinate in your states may help, or have goal-conditioned value functions. In addition, make sure your reward shaping makes the environment actually solvable!
Comments.UserDisplayName: Luca Anzalone
Tags: <reinforcement-learning><policies><goal-based-agents>
Title: What is the best strategy to train a model with multi (sub)goals in the same environment?
UserDisplayName: Dave
ThreadText: # What is the best strategy to train a model with multi (sub)goals in the same environment?
  
  (Asked by: username_0 on 2024-07-17)
  
  To be able to explain my question I thought it is probably better to consider the following example: Let's take an environment, where a bridge crane need to lift a barrel from the position "start" and move it to the position "goal" moving along the axes: $X, Y, Z$.
  
  [![example](https://i.sstatic.net/YvdRVMx7.jpg)](https://i.sstatic.net/YvdRVMx7.jpg)
  
  But the movement cannot be straight to the goal. The movement from the start to the goal is based on the presence of obstable in the enviroment and some pratical rules (e.g. the barrel must be lifted first, then horizontally moved and at the end dropped at the goal position).
  
  I'm struggling to figure out, how to train an agent to be able to accomplish such a task.
  But here I want to share my thoughts:
  
  1. At first I thought to use PPO or SAC as model free algorithms and assigning different *rewards*, when reached the goal and the positions in between (let's call them *waypoints*) In other words, the agent starts from the *start* position and gets a reward of +100 points if the agent reaches $a$ in some time. Now... in order to move forward I would need to give a higher reward to $b$. But that means, that the agent will try during the training to reach $b$ directly instead going to $b$ over $a$.
  2. Even if the issues in the first point are somehow solvable there is the problem, that the agent would learn to reach the goal through the waypoints seen during the training, even if the new environment is now different (obstacles are in different position, the waypoints changed their positions, etc.). So the agent would learn "memorizing" the training environment, without generalization.
  
  What could be the best strategy in such enviroments?
  Thanks
  
  **Tags:** reinforcement-learning, policies, goal-based-agents
  > **Comments:**
  > **username_1**: Have you tried to combine PPO/SAC (or similar) with *Hindsight Experience Replay*? HER allows to train goal-based policies. Maybe you can split the problem in subgoals, having also a goal-dependent reward function. For the generalization issue you may need to apply *domain randomization* and/or extend the state representation to include information about the obstacles in the environment. Also,I think that randomizing the start/goal positions as well the obstacles during training should help  
  > **username_0**: There are many things to unpack here: 1) PPO is onpolicy HER is for offpolicy algorihtms. So it could fit at best with SAC 2) Yes I tried SAC + HER: it miserably fails. The reason is that in case of HER I need a sparse reward (e.g. +1 if the (sub) goal is reached, 0 otherwise). Since the parameter space is continuous there are too many states to be explored, it simply does not work. For this reason I switched back to shaping the reward instead using a sparsed one  
  > **username_0**: 3) You say. "split the problem in subgoals". But this is actually the point. How? Should I train the agent to move first up. And when it does it, it should learns to move horizontally? 4) Domain randomization is what I'm doing now. The result is that the agent does not converge to a solution  
  > **username_1**: Maybe you can try to train multiple policies (agents), and then coordinate them by hand-made rules: you can have an agent that lifts the barrel up, then another specialized for moving it horizontally, and eventually another than puts the barrel on the ground. Maybe this can help. Also including the goal's coordinate in your states may help, or have goal-conditioned value functions. In addition, make sure your reward shaping makes the environment actually solvable!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-07-25. Score: 2 | Accepted answer
  
  I do not have enough reputation to just comment, but here are my thoughts:
  
  Do you actually need to use subgoals/waypoints?
  If your objective is simply how to train an agent to be able to accomplish such a task, then I am thinking you do not need to break it into subgoals/waypoints.
  Indeed, the environment you describe can be reduced to a 2d environment with a starting position and a single goal position, where the agent must find the shortest path between these two positions. Since the barrel must be lifted first whatever happens, this can be hard coded in the reset point (I don't think there is much use in training a RL agent to learn to just lift an object as there are not many ways this can be done!), same for the drop at the goal position (simply set the routine for the drop once the agent has reached the goal position).
  
  If you actually need waypoints, then as @Luca Anzalone suggested in the comments, you can still include the goal's coordinate in your states.
  
  For Domain Randomization, if the agent does not observe the positions (and size?) of the obstacles, then the policy will not be able to know where it can move (this is due to the overly-cautious or "conservative" nature of DR policies). The only way this would work is if your obstacles are distributed such that there is one specific path that is always opened. Otherwise, you should consider adding the obstacle information to your states (the agent does not need to observe the full space, but it should be able to at least observe when it is close to an obstacle, so that it can avoid it or move away).
  
  If you really need to train the policy for lifting and dropping, then you could take a look at continual reinforcement learning (specifically Multi-task Learning?), but I think that even in this case, a properly designed reward function could be enough (one that penalizes the agent for starting to move while the barrel is still on the ground [of course this means a 3d environment instead of the 2d case I mentioned above] - the penalty can be proportional to the height of the barrel and maybe to the distance from the starting or goal position).
  
  > **Comments:**
  > **username_0**: Many thanks to your interesting answer/comment. But the problem is that, as you pointed out, the reward function will be a sum of many different factors, which in worst case work against each other. So the barrel could move perfectly on the vertical but getting a penalty because it moves far away from the final goal (e.g. the goal is at the same level as the start position)  
  > **username_2**: @Dave Yes, reward shaping is well known problem in RL... There isn't many ways to solve the issue. Have you considered mutli-task reinforcement learning? Another possibility for training a single policy is multi-objective reinforcement learning in which the task reward and penalty are weighted by some preference vector. In particular, take a look at dynamics weight MORL algorithms: [link](https://proceedings.mlr.press/v97/abels19a). The paper is for Q-learning, but you can adapt it to SAC or PPO by simply using universal policy networks.  
  > **username_0**: Your intuition is correct man. But I still struggle in understanding the concept. That a scalar reward too less for complex robot system is, is clear. I read the paper carefully and looking for some simple tutorial on its implementation. But yeah, that's probably the way to go  
  
  ---
  
  