Answers.Body: <blockquote>
  <p>Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?</p>
  </blockquote>
  <p>Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and reinforcement learning (RL) methods.</p>
  <p>Although it is possible to accumulate matrices in various ways, by e.g. simple matrix addition, and come up with an analog for <em>expected return</em> which would be a weighted sum of matrices, you then get stuck. There is no fixed way to rank matrices and decide whether one is a better result than another. This is a requirement for any learning process that aims to improve at a task - it needs feedback that changes it makes are better or worse related to some reference. As a result, most objective functions and metrics in optimisation use real-valued scalars, which can always be placed into order to decide a highest or lowest value.</p>
  <p>This does not prevent you using a matrix representation for your project, if it is a natural fit. To turn it into a usable reward, you will need to convert that matrix into a real-valued metric. Perhaps the <a href="https://en.wikipedia.org/wiki/Matrix_norm" rel="nofollow noreferrer">L2 norm</a> or other standard measure that summarises the matrix will be good for your task.</p>
  <p>It is possible to process multiple scalar rewards at once with single learner, using <a href="https://arxiv.org/abs/1908.08342" rel="nofollow noreferrer">multi-objective reinforcement learning</a>. Applied to your problem, this would give you access to a matrix of policies, each of which maximised the reward value of one cell within the matrix. It also allows for switching between objectives in a hierarchical manner using a &quot;policy of policies&quot; if some preference for what to achieve changes. It is not 100% clear, but I do not think this is what you want to do.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-02T22:35:00.730Z
Answers.Text: It might be worthing explicitly mentioning in your answer "multi-objective reinforcement learning" and the ["reward hypothesis"](https://ai.stackexchange.com/q/16610/2444). [This survey](https://www.jair.org/index.php/jair/article/view/10836/25862) also seems a good reference for further details on MORL (i.e. it formulates the MDPs, value functions, etc., in the case of vector-valued rewards).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-03T07:19:33.867Z
Answers.Text: @nbro: Thanks, I'll have athink about that. In this case MORL is a bit of a red herring as OP appears to have a singular task. If I have it right, the vector of rewards in MORL represents multiple separate goals to be processed at once, such that a higher order goal selector policy can be used in a hierarchical fashion. It is not a way to avoid needing scalar reward values for optimality, but a way to parallel process multiple scalar reward schemes in an otherwise identical environment.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-02T17:16:25.873Z
Answers.LastActivityDate: 2021-02-03T07:30:29.787Z
Answers.LastEditDate: 2021-02-03T07:30:29.787Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working with images, but I'd like to observe the agent doing it in real-time. Just a fun side project.</p>
  <p>Anyway, to encourage an RL agent to craft a specific image I'm crafting a reward function that returns a <span class="math-container">$N \times N$</span> dimensional matrix. Which represents the distance between the state of the target image (RGB values for each pixel location) and the image the agent crafted.</p>
  <p>Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T16:00:01.610Z
Comments.Text: How would you adapt [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) to work with vectorized rewards? Note that I changed the title to be what I think is your actual question. If that's not your actual question, let us know what it is.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T22:28:38.020Z
Comments.Text: See also [this](https://ai.stackexchange.com/q/16610/2444) and [this](https://ai.stackexchange.com/q/22900/2444).
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><dqn><reward-functions><reward-design><multi-objective-rl>
Title: Can the rewards be matrices when using DQN?
UserDisplayName: junfanbl
ThreadText: # Can the rewards be matrices when using DQN?
  
  (Asked by: username_0 on 2021-02-02)
  
  I have a basic question. I'm working towards developing a reward function for my DQN. I'd like to train an RL agent to edit pixels on an image. I understand that convolutions are ideal for working with images, but I'd like to observe the agent doing it in real-time. Just a fun side project.
  
  Anyway, to encourage an RL agent to craft a specific image I'm crafting a reward function that returns a $N \times N$ dimensional matrix. Which represents the distance between the state of the target image (RGB values for each pixel location) and the image the agent crafted.
  
  Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?
  
  **Tags:** reinforcement-learning, dqn, reward-functions, reward-design, multi-objective-rl
  
  > **Comments:**
  > **username_1**: How would you adapt [DQN](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf) to work with vectorized rewards? Note that I changed the title to be what I think is your actual question. If that's not your actual question, let us know what it is.  
  > **username_1**: See also [this](https://ai.stackexchange.com/q/16610/2444) and [this](https://ai.stackexchange.com/q/22900/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-02-02. Score: 3 | Accepted answer
  
  > Generally speaking, is it better for rewards to be a scalar, or is using matrices okay?
  
  Rewards need to be scalar, real values to match to standard theory of Markov decision processes (MDPs) and reinforcement learning (RL) methods.
  
  Although it is possible to accumulate matrices in various ways, by e.g. simple matrix addition, and come up with an analog for *expected return* which would be a weighted sum of matrices, you then get stuck. There is no fixed way to rank matrices and decide whether one is a better result than another. This is a requirement for any learning process that aims to improve at a task - it needs feedback that changes it makes are better or worse related to some reference. As a result, most objective functions and metrics in optimisation use real-valued scalars, which can always be placed into order to decide a highest or lowest value.
  
  This does not prevent you using a matrix representation for your project, if it is a natural fit. To turn it into a usable reward, you will need to convert that matrix into a real-valued metric. Perhaps the [L2 norm](https://en.wikipedia.org/wiki/Matrix_norm) or other standard measure that summarises the matrix will be good for your task.
  
  It is possible to process multiple scalar rewards at once with single learner, using [multi-objective reinforcement learning](https://arxiv.org/abs/1908.08342). Applied to your problem, this would give you access to a matrix of policies, each of which maximised the reward value of one cell within the matrix. It also allows for switching between objectives in a hierarchical manner using a "policy of policies" if some preference for what to achieve changes. It is not 100% clear, but I do not think this is what you want to do.
  
  > **Comments:**
  > **username_1**: It might be worthing explicitly mentioning in your answer "multi-objective reinforcement learning" and the ["reward hypothesis"](https://ai.stackexchange.com/q/16610/2444). [This survey](https://www.jair.org/index.php/jair/article/view/10836/25862) also seems a good reference for further details on MORL (i.e. it formulates the MDPs, value functions, etc., in the case of vector-valued rewards).  
  > **username_2**: @username_1: Thanks, I'll have athink about that. In this case MORL is a bit of a red herring as OP appears to have a singular task. If I have it right, the vector of rewards in MORL represents multiple separate goals to be processed at once, such that a higher order goal selector policy can be used in a hierarchical fashion. It is not a way to avoid needing scalar reward values for optimality, but a way to parallel process multiple scalar reward schemes in an otherwise identical environment.  
  
  ---
  
  