Answers.Body: <p>The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step.</p>
  <p><span class="math-container">$(o^a_t,m^a′_{t−1},u^a_{t−1},a)$</span> (input) <span class="math-container">$\rightarrow$</span> input network (embedding) <span class="math-container">$\rightarrow$</span> <span class="math-container">$z_t$</span> (task representation)</p>
  <hr />
  <p>According to to section 6.1 of the paper, the input is a tuple represented as <span class="math-container">$(o^a_t,m^a′_{t−1},u^a_{t−1},a)$</span>. Each of these  terms is described in sec 3 as:</p>
  <ul>
  <li><p><span class="math-container">$o_t$</span> - The observation. The authors assume a <a href="https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process" rel="nofollow noreferrer">POMDP</a></p>
  </li>
  <li><p>Two types of actions:</p>
  <ul>
  <li><span class="math-container">$u_t$</span> - An environment action selects by all the agents at each time step in exchange for a team reward</li>
  <li><span class="math-container">$m_t$</span> - A communication action, observed by other agent but has no direct impact on the reward or env</li>
  </ul>
  </li>
  <li><p><span class="math-container">$a$</span> - The agent (This being a multi-agent DQN algorithm)</p>
  </li>
  </ul>
  <p>They form the input to the Deep Recurrent Q-Network architecture.</p>
  <p>The purpose of the embedding network is to receive a tuple of these inputs and produce a state
  embedding <span class="math-container">$z$</span>. This state embedding is then fed  into a hidden state <span class="math-container">$h^a_{t-1}$</span> of the RNN.</p>
  <p>Though the authors refer to it as an input network that produces embedding, in practice, they use different embedding functions for each of these inputs. The final task/state embedding <span class="math-container">$z_t^a$</span> is expressed as a sum:</p>
  <p><span class="math-container">$$
  z_t^a = \text{MLP}(o_t^a) + \text{MLP}(m_{t-1}) + \text{LookupTable}(u_{t-1}) + \text{LookupTable}(a)
  $$</span></p>
  <p>The summed embeddings are all of the same size.</p>
  <p>This answer makes an assumption that by &quot;embedding layer&quot; you meant the input embedding network. <em>Since the paper makes no reference to a single embedding layer in the model architecture.</em></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-22T14:52:08.670Z
Answers.LastActivityDate: 2020-11-22T14:58:57.960Z
Answers.LastEditDate: 2020-11-22T14:58:57.960Z
Answers.UserDisplayName: mugoh
Body: <p>When describing the model architecture for a deep recurrent q network, the authors of the paper <a href="https://arxiv.org/pdf/1605.06676.pdf" rel="nofollow noreferrer">Learning to Communicate with Deep Multi-Agent Reinforcement Learning</a></p>
  <blockquote>
  <p>each agent consists of a recurrent neural network (RNN), unrolled for <span class="math-container">$T$</span> time-steps, that maintains an internal state <span class="math-container">$h$</span>, an input network for producing a task embedding <span class="math-container">$z$</span>, and an output network for the Q-values and the messages <span class="math-container">$m$</span>. The input for agent <span class="math-container">$a$</span> is defined as a tuple of <span class="math-container">$\left(o_{t}^{a}, m_{t-1}^{a^{\prime}}, u_{t-1}^{a}, a\right)$</span>.</p>
  </blockquote>
  <p>Can someone explain what the purpose of the embedding layer is in this specific context?</p>
  <p>Implementation can be found <a href="https://colab.research.google.com/gist/MJ10/2c0d1972f3dd1edcc3cd17c636aac8d2/dial.ipynb" rel="nofollow noreferrer">here</a>.</p>
  
Tags: <reinforcement-learning><recurrent-neural-networks><dqn><deep-rl><embeddings>
Title: What is the role of embeddings in a deep recurrent Q network?
UserDisplayName: kikram
ThreadText: # What is the role of embeddings in a deep recurrent Q network?
  
  (Asked by: username_0 on 2020-07-03)
  
  When describing the model architecture for a deep recurrent q network, the authors of the paper [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://arxiv.org/pdf/1605.06676.pdf)
  
  > each agent consists of a recurrent neural network (RNN), unrolled for $T$ time-steps, that maintains an internal state $h$, an input network for producing a task embedding $z$, and an output network for the Q-values and the messages $m$. The input for agent $a$ is defined as a tuple of $\left(o\_{t}^{a}, m\_{t-1}^{a^{\prime}}, u\_{t-1}^{a}, a\right)$.
  
  Can someone explain what the purpose of the embedding layer is in this specific context?
  
  Implementation can be found [here](https://colab.research.google.com/gist/MJ10/2c0d1972f3dd1edcc3cd17c636aac8d2/dial.ipynb).
  
  **Tags:** reinforcement-learning, recurrent-neural-networks, dqn, deep-rl, embeddings
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-11-22. Score: 2 | Accepted answer
  
  The purpose of the input network is to embed the input tuple into a state/task representation, that can then be fed into the RNN hidden state at each time step.
  
  $(o^a\_t,m^a′\_{t−1},u^a\_{t−1},a)$ (input) $\rightarrow$ input network (embedding) $\rightarrow$ $z\_t$ (task representation)
  
  ---
  
  According to to section 6.1 of the paper, the input is a tuple represented as $(o^a\_t,m^a′\_{t−1},u^a\_{t−1},a)$. Each of these terms is described in sec 3 as:
  
  * $o\_t$ - The observation. The authors assume a [POMDP](https://en.wikipedia.org/wiki/Partially_observable_Markov_decision_process)
  * Two types of actions:
  
    + $u\_t$ - An environment action selects by all the agents at each time step in exchange for a team reward
    + $m\_t$ - A communication action, observed by other agent but has no direct impact on the reward or env
  * $a$ - The agent (This being a multi-agent DQN algorithm)
  
  They form the input to the Deep Recurrent Q-Network architecture.
  
  The purpose of the embedding network is to receive a tuple of these inputs and produce a state
  embedding $z$. This state embedding is then fed into a hidden state $h^a\_{t-1}$ of the RNN.
  
  Though the authors refer to it as an input network that produces embedding, in practice, they use different embedding functions for each of these inputs. The final task/state embedding $z\_t^a$ is expressed as a sum:
  
  $$
  z\_t^a = \text{MLP}(o\_t^a) + \text{MLP}(m\_{t-1}) + \text{LookupTable}(u\_{t-1}) + \text{LookupTable}(a)
  $$
  
  The summed embeddings are all of the same size.
  
  This answer makes an assumption that by "embedding layer" you meant the input embedding network. *Since the paper makes no reference to a single embedding layer in the model architecture.*
  
  
  ---
  
  