Answers.Body: <p>Yes, we can do that (look at the examples from MORL Gymnasium). They have a linear function to weight more rewards into one scalar and they use only this value as any single policy single reward problem out there.</p>
  <p>But...</p>
  <p>But let's take the <a href="https://www.youtube.com/watch?v=ma6fbvy77Uo&amp;list=PLOkmXPXHDP22o_VQTBGQXTJp-8Gc8FtCu" rel="nofollow noreferrer">panda kitchen environment</a>, where a robotic arm learns how to open a microwave or to lift and tilt a boiler or to turn the right temperature on a stove or how to grasp a glass and so. Everything in the same environment.
  With conventional RL you can teach the robotic arm to perform only one of those tasks. But how to train the robot to open the microwave and put a glass in it? Or to put some water in a boiler and then set the temperature?
  You need a &quot;supervisor&quot; policy, which learn transitions between those tasks and learns the sequence of tasks to perform in order to increase the overall reward.</p>
  <p>Look at the playlist I linked above. It will answer many questions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-07T07:16:16.973Z
Answers.LastActivityDate: 2024-09-07T07:16:16.973Z
Answers.UserDisplayName: Dave
Body: <p>I have recently posted a question here about a problem that I have controlling a robotic arm.
  Basically I have a dense reward for the arms position, and a sparse reward for the arms stiffness: <a href="https://ai.stackexchange.com/questions/45049/reward-shaping-for-dense-and-sparse-rewards?noredirect=1#comment65807_45049">Reward shaping for dense and sparse rewards</a></p>
  <p>I am using PPO at the moment in my attempts to solve it but with little success. Now I learned about specialised Multi-Objective RL algorithms and think that they might be suited better.</p>
  <p>Still I am wondering why that would be or why we would need a separate algorithm for MOO.
  <strong>Can we not always put all reward for multiple objectives into some formula to form a single reward?</strong>
  Weighted sum, ratio, difference,  whatever?</p>
  
Tags: <reinforcement-learning><optimization><multi-objective-rl>
Title: Why does Multi Objective RL exist?
UserDisplayName: mavex857
ThreadText: # Why does Multi Objective RL exist?
  
  (Asked by: username_0 on 2024-03-10)
  
  I have recently posted a question here about a problem that I have controlling a robotic arm.
  Basically I have a dense reward for the arms position, and a sparse reward for the arms stiffness: [Reward shaping for dense and sparse rewards](https://ai.stackexchange.com/questions/45049/reward-shaping-for-dense-and-sparse-rewards?noredirect=1#comment65807_45049)
  
  I am using PPO at the moment in my attempts to solve it but with little success. Now I learned about specialised Multi-Objective RL algorithms and think that they might be suited better.
  
  Still I am wondering why that would be or why we would need a separate algorithm for MOO.
  **Can we not always put all reward for multiple objectives into some formula to form a single reward?**
  Weighted sum, ratio, difference, whatever?
  
  **Tags:** reinforcement-learning, optimization, multi-objective-rl
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-07. Score: 0
  
  Yes, we can do that (look at the examples from MORL Gymnasium). They have a linear function to weight more rewards into one scalar and they use only this value as any single policy single reward problem out there.
  
  But...
  
  But let's take the [panda kitchen environment](https://www.youtube.com/watch?v=ma6fbvy77Uo&list=PLOkmXPXHDP22o_VQTBGQXTJp-8Gc8FtCu), where a robotic arm learns how to open a microwave or to lift and tilt a boiler or to turn the right temperature on a stove or how to grasp a glass and so. Everything in the same environment.
  With conventional RL you can teach the robotic arm to perform only one of those tasks. But how to train the robot to open the microwave and put a glass in it? Or to put some water in a boiler and then set the temperature?
  You need a "supervisor" policy, which learn transitions between those tasks and learns the sequence of tasks to perform in order to increase the overall reward.
  
  Look at the playlist I linked above. It will answer many questions.
  
  
  ---
  
  