Body: <p>I have a Reinforcement Learning environment where the state is a 2D matrix with 0s and 1s (only one column with the value of 1 in each row).</p>
  <p>Example:</p>
  <pre><code>(
   (0, 1, 0),
   (0, 0, 1),
   (1, 0, 0),
   (0, 0, 0),
   (0, 1, 0)
  )
  </code></pre>
  <p>The action the agent must take is for each row in the input, choose one resource out of 12 resources the agent has if there is a column with the value of 1 in that row, else choose no resource if the row has 0s only (example: <code>row[3]</code> wouldn't have any resources chosen for it by the agent). The rows correspond to the users the agent must allocate resources to.</p>
  <p>In the step() method in the RL environment, the agent would receive a reward or a penalty depending on the action. If the reward is positive, the agent updates the state matrix, putting a 0 instead of 1 in the rows corresponding to the users that were allocated resources, which should be the next state. If the reward is negative, the episode ends, the environment resets and a new state is received by the agent</p>
  <p>It came to my understanding that, in a deep learning approach, the DQN agent would receive a 2D matrix of 0s and 1s as input to its neural network (the state matrix), and output a vector with the chosen resources for each row of the input.</p>
  <p>The network must choose a resource out of 12 resources for each row if that row has a 1 in it, and no resource is chosen if there is no column with the value of 1 in that row of the input. In other words, the network must choose an element out of 12 and output a vector with the chosen elements, depending on the input matrix.</p>
  <p>Is there a way to do this using Deep Q-Learning and neural networks ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T18:51:43.823Z
Comments.Text: I am having trouble picturing this - could you give an example so that it is clearer what " only one column with the value of 1 in each row" means, and also what the action selection looks like? Am I right in thinking there must be more rows than columns in this problem (or at least as many)? Also you should explain the consequence of the choice - is this to optimise some measure of success/results from the action vector, and if so what happens next?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T18:55:33.130Z
Comments.Text: For Q-learning to apply, two things should happen - 1) there needs to be some measure of usefulness or success, and 2) the input (state) should repeat with a variation, and the possible variations depend somehow on the action vector. It is not at all clear what these things might be from your description
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T19:11:45.887Z
Comments.Text: @NeilSlater thank you for your comment, I edited my post. I hope it's better explained. Answering your questions, yes there are more rows than columns. In the step() method in the environment, the action choice is rewarded according to the constraints I have defined.
Comments.UserDisplayName: Ness
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T19:21:37.050Z
Comments.Text: Thank you Ness, that is much clearer. I am still not sure what the purpose is though. Assuming for example that the agent chose (1, 5, 7, 0, 12) for allocating resources (where 0 is "no resource"), what would happen next? It is very important for choosing DQN and RL in general over some other algorithm: Will the values in the next matrix presented to the agent be in any way dependent on what the agent chose? Or another way to put things - are you trying to optimise a single function, or is there a time element with the goal of getting the best total result over multiple related inputs?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T19:27:48.537Z
Comments.Text: @NeilSlater originally, in my Q-learning agent, the action is another matrix of 0s and 1s, so row[0][1] = 1 would mean that resource0 was allocated to user1. But I got stuck trying to figure out how to do that. So, I had the idea of having the network choose from the resources. So, for example, the vector would have 10 elements (because we have 10 users (equals number of rows in the state matrix)), vector[1] = 5 would mean that resource5 was allocated to user1. That would help with the reward calculation later. I hope I answered your question.
Comments.UserDisplayName: Ness
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T19:33:50.700Z
Comments.Text: It should be possible to have your original idea for an output. But that is not what what I was asking about. I am asking what happens after resource5 is allocated to user1 etc, and after you have worked out the reward. What happens next? Is there any concept of "next" here at all, or is the process you are trying to optimise a series of separate events each of which will be separately optimised according to the same kind of inputs and outputs?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T19:40:20.777Z
Comments.Text: @NeilSlater Hmm, I must say I'm a beginner at Reinforcement Learning, so I will try to answer you to the best of my ability. If the reward is positive, the agent updates the state matrix, putting a 0 instead of 1 in the rows corresponding to the users that were allocated resources, which should be the next state. If the reward is negative, the episode ends, the environment resets and a new state is received by the agent. The state wouldn't be exactly the same in each episode, the position of the 1 and 0s in each row would vary, hence the output would differ as well depending on the state.
Comments.UserDisplayName: Ness
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-04T19:43:06.003Z
Comments.Text: OK, that is clearer. I suggest add this part to the question body, it is important: "If the reward is positive, the agent updates the state matrix, putting a 0 instead of 1 in the rows corresponding to the users that were allocated resources, which should be the next state. If the reward is negative, the episode ends, the environment resets and a new state is received by the agent". To me, this clearly describes a problem where reinforcement learning could be applied.
Comments.UserDisplayName: Neil Slater
Tags: <deep-learning><tensorflow><keras><deep-rl><deep-neural-networks>
Title: DQN Agent with a 2D matrix as input in Keras
UserDisplayName: Ness
ThreadText: # DQN Agent with a 2D matrix as input in Keras
  
  (Asked by: username_0 on 2020-12-04)
  
  I have a Reinforcement Learning environment where the state is a 2D matrix with 0s and 1s (only one column with the value of 1 in each row).
  
  Example:
  
  ```
  (
   (0, 1, 0),
   (0, 0, 1),
   (1, 0, 0),
   (0, 0, 0),
   (0, 1, 0)
  )
  
  ```
  
  The action the agent must take is for each row in the input, choose one resource out of 12 resources the agent has if there is a column with the value of 1 in that row, else choose no resource if the row has 0s only (example: `row[3]` wouldn't have any resources chosen for it by the agent). The rows correspond to the users the agent must allocate resources to.
  
  In the step() method in the RL environment, the agent would receive a reward or a penalty depending on the action. If the reward is positive, the agent updates the state matrix, putting a 0 instead of 1 in the rows corresponding to the users that were allocated resources, which should be the next state. If the reward is negative, the episode ends, the environment resets and a new state is received by the agent
  
  It came to my understanding that, in a deep learning approach, the DQN agent would receive a 2D matrix of 0s and 1s as input to its neural network (the state matrix), and output a vector with the chosen resources for each row of the input.
  
  The network must choose a resource out of 12 resources for each row if that row has a 1 in it, and no resource is chosen if there is no column with the value of 1 in that row of the input. In other words, the network must choose an element out of 12 and output a vector with the chosen elements, depending on the input matrix.
  
  Is there a way to do this using Deep Q-Learning and neural networks ?
  
  **Tags:** deep-learning, tensorflow, keras, deep-rl, deep-neural-networks
  
  > **Comments:**
  > **username_1**: I am having trouble picturing this - could you give an example so that it is clearer what " only one column with the value of 1 in each row" means, and also what the action selection looks like? Am I right in thinking there must be more rows than columns in this problem (or at least as many)? Also you should explain the consequence of the choice - is this to optimise some measure of success/results from the action vector, and if so what happens next?  
  > **username_1**: For Q-learning to apply, two things should happen - 1) there needs to be some measure of usefulness or success, and 2) the input (state) should repeat with a variation, and the possible variations depend somehow on the action vector. It is not at all clear what these things might be from your description  
  > **username_0**: @username_1 thank you for your comment, I edited my post. I hope it's better explained. Answering your questions, yes there are more rows than columns. In the step() method in the environment, the action choice is rewarded according to the constraints I have defined.  
  > **username_1**: Thank you username_0, that is much clearer. I am still not sure what the purpose is though. Assuming for example that the agent chose (1, 5, 7, 0, 12) for allocating resources (where 0 is "no resource"), what would happen next? It is very important for choosing DQN and RL in general over some other algorithm: Will the values in the next matrix presented to the agent be in any way dependent on what the agent chose? Or another way to put things - are you trying to optimise a single function, or is there a time element with the goal of getting the best total result over multiple related inputs?  
  > **username_0**: @username_1 originally, in my Q-learning agent, the action is another matrix of 0s and 1s, so row[0][1] = 1 would mean that resource0 was allocated to user1. But I got stuck trying to figure out how to do that. So, I had the idea of having the network choose from the resources. So, for example, the vector would have 10 elements (because we have 10 users (equals number of rows in the state matrix)), vector[1] = 5 would mean that resource5 was allocated to user1. That would help with the reward calculation later. I hope I answered your question.  
  > **username_1**: It should be possible to have your original idea for an output. But that is not what what I was asking about. I am asking what happens after resource5 is allocated to user1 etc, and after you have worked out the reward. What happens next? Is there any concept of "next" here at all, or is the process you are trying to optimise a series of separate events each of which will be separately optimised according to the same kind of inputs and outputs?  
  > **username_0**: @username_1 Hmm, I must say I'm a beginner at Reinforcement Learning, so I will try to answer you to the best of my ability. If the reward is positive, the agent updates the state matrix, putting a 0 instead of 1 in the rows corresponding to the users that were allocated resources, which should be the next state. If the reward is negative, the episode ends, the environment resets and a new state is received by the agent. The state wouldn't be exactly the same in each episode, the position of the 1 and 0s in each row would vary, hence the output would differ as well depending on the state.  
  > **username_1**: OK, that is clearer. I suggest add this part to the question body, it is important: "If the reward is positive, the agent updates the state matrix, putting a 0 instead of 1 in the rows corresponding to the users that were allocated resources, which should be the next state. If the reward is negative, the episode ends, the environment resets and a new state is received by the agent". To me, this clearly describes a problem where reinforcement learning could be applied.  
  
  ---
  
  ## 0 answers
  
  