Answers.Body: <p>This is not necessarily the only way to do this but it would be the approach I'd take.</p>
  <p>Assuming your agents position is a vector in <span class="math-container">$\mathbb{R}^d$</span>, then I would have the network take as input this position vector and pass it through a fully connected layer. I would also take as input the matrix and pass it through a convolutional layer(s) and flatten the output so it is now also a vector in <span class="math-container">$\mathbb{R}^{d'}$</span>. I would then concatenate these together so you have a vector in <span class="math-container">$\mathbb{R}^{d + d'}$</span> and pass this through some fully connected layers as usual.</p>
  <p>As for <em>how many</em> layers and which activation to use this is something you'll have to do by trial and error as this really is problem specific.</p>
  <p>Your output would typically be a score for each of the action combinations, though as one of your outputs is a matrix this could potentially make things expensive to compute as you would need <span class="math-container">$2^{n \times m}$</span> binary representations just for the matrix (<span class="math-container">$n$</span> is the number of rows and <span class="math-container">$m$</span> is the number of columns).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T11:56:19.257Z
Answers.LastActivityDate: 2020-11-18T11:56:19.257Z
Answers.UserDisplayName: David
Body: <p>I have built my own RL environment, where a state is composed of two elements: the agent's position and a matrix of 0s and 1s (1 if a user has requested a service from the agent, 0 otherwise); an action is composed of 3 elements: the movement the agent chooses (up, down, left or right), a matrix of 0s and 1s (1 if a resource has been allocated to a user, 0 otherwise), and a vector representing the allocation of another type of resource (the vector contains the values allocated to the users).</p>
  <p>I am currently trying to build a Deep Q Learning agent, I am a bit confused however as to what model (example Sequential), what type of layers (example Dense layers), how many layers, what activation mode I should use, and what the state and action sizes are. (Taking this code as a reference <a href="https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/cartpole_dqn.ipynb" rel="nofollow noreferrer">cartpole dqn agent</a>)</p>
  <p>I also do not know what my inputs and outputs should be.</p>
  <p>The examples I have come across are rather simple and I don't know how to approach setting it all up for my agent.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T21:14:19.220Z
Comments.Text: assuming your agents position is a vector in $\mathbb{R}^d$, you could have a network that takes as input the position and passes it through a fully connected layer, the matrix passed through a convolutional layer, and the concatenate the two outputs of those layers to pass through further fully connected layers.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-12T21:37:32.517Z
Comments.Text: Genius ! Thank you so much for your answer ^^
Comments.UserDisplayName: Ness
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-18T11:43:15.383Z
Comments.Text: I'll add my comment as an answer since you seem happy with it. :-)
Comments.UserDisplayName: David
Tags: <deep-learning><python><keras><deep-rl><deep-neural-networks>
Title: DQN layers when state space and action space are multi dimensional
UserDisplayName: Ness
ThreadText: # DQN layers when state space and action space are multi dimensional
  
  (Asked by: username_0 on 2020-11-12)
  
  I have built my own RL environment, where a state is composed of two elements: the agent's position and a matrix of 0s and 1s (1 if a user has requested a service from the agent, 0 otherwise); an action is composed of 3 elements: the movement the agent chooses (up, down, left or right), a matrix of 0s and 1s (1 if a resource has been allocated to a user, 0 otherwise), and a vector representing the allocation of another type of resource (the vector contains the values allocated to the users).
  
  I am currently trying to build a Deep Q Learning agent, I am a bit confused however as to what model (example Sequential), what type of layers (example Dense layers), how many layers, what activation mode I should use, and what the state and action sizes are. (Taking this code as a reference [cartpole dqn agent](https://github.com/the-deep-learners/TensorFlow-LiveLessons/blob/master/notebooks/cartpole_dqn.ipynb))
  
  I also do not know what my inputs and outputs should be.
  
  The examples I have come across are rather simple and I don't know how to approach setting it all up for my agent.
  
  **Tags:** deep-learning, python, keras, deep-rl, deep-neural-networks
  > **Comments:**
  > **username_1**: assuming your agents position is a vector in $\mathbb{R}^d$, you could have a network that takes as input the position and passes it through a fully connected layer, the matrix passed through a convolutional layer, and the concatenate the two outputs of those layers to pass through further fully connected layers.  
  > **username_0**: Genius ! Thank you so much for your answer ^^  
  > **username_1**: I'll add my comment as an answer since you seem happy with it. :-)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-18. Score: 1 | Accepted answer
  
  This is not necessarily the only way to do this but it would be the approach I'd take.
  
  Assuming your agents position is a vector in $\mathbb{R}^d$, then I would have the network take as input this position vector and pass it through a fully connected layer. I would also take as input the matrix and pass it through a convolutional layer(s) and flatten the output so it is now also a vector in $\mathbb{R}^{d'}$. I would then concatenate these together so you have a vector in $\mathbb{R}^{d + d'}$ and pass this through some fully connected layers as usual.
  
  As for *how many* layers and which activation to use this is something you'll have to do by trial and error as this really is problem specific.
  
  Your output would typically be a score for each of the action combinations, though as one of your outputs is a matrix this could potentially make things expensive to compute as you would need $2^{n \times m}$ binary representations just for the matrix ($n$ is the number of rows and $m$ is the number of columns).
  
  
  ---
  
  