Answers.Body: <p>Yes, I would say more, that hidden state can be a tensor of arbitrary dimensionality. For vanilla RNN the update rules of the hidden state and the output are:
  <span class="math-container">$$
  h_t = \sigma_h(W_h x_t + U_h h_{t-1} + b_h) 
  $$</span>
  <span class="math-container">$$
  y_t = \sigma_y(W_y h_t + b_y)
  $$</span>
  Here <span class="math-container">$W_h$</span> is input to hidden state matrix, and <span class="math-container">$U_h$</span> is hidden state to hidden state, <span class="math-container">$W_y$</span> is hidden state to output. One can simply upgrade these matrix products to more general case of tensor products, such that they could handle multidimensional input data and hidden state.</p>
  <p>The contraction of the <span class="math-container">$m$</span>-dimensional tensor <span class="math-container">$X$</span> with <span class="math-container">$n$</span>-dimensional Y (we assume <span class="math-container">$m \geq n$</span>) by maximal amount of indices will be <span class="math-container">$m-n$</span> dimensional tensor <span class="math-container">$Z$</span>:
  <span class="math-container">$$
  X_{i_1 \ldots i_m} Y_{i_1 \ldots i_n} = Z_{i_{n+1} \ldots i_m}
  $$</span></p>
  <p>Namely, imagine, that the <span class="math-container">$x_t = (x_t)_{i_1 \ldots i_X}$</span> is an <span class="math-container">$d_x$</span>-dimensional tensor. Then take <span class="math-container">$W_h$</span> to be  <span class="math-container">$d_{wh}$</span> dimensional tensor, hidden state <span class="math-container">$h$</span> to be <span class="math-container">$d_h = d_{wh} - d_{x}$</span> -dimensional, <span class="math-container">$U_h$</span> is <span class="math-container">$d_{uh}$</span> - dimensional, <span class="math-container">$W_y$</span> is <span class="math-container">$d_{wy}$</span> -dimensional. The output <span class="math-container">$y_t$</span> of the RNN will be <span class="math-container">$d_{wy} - d_{h}$</span> dimensional tensor.</p>
  <p>Here note, that in order for the sum to be make sense in the first equation, dimensions need to match:
  <span class="math-container">$$
  d_{uh} - d_{h} = d_{h}
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-20T07:55:15.413Z
Answers.LastActivityDate: 2021-02-20T07:55:15.413Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>The question is, why you specifically want a matrix? I assume you mean per-frame features. In such case you can use a ConvNet as a feature extractor, i.e. it outputs a vector of features of fixed size. This vector is an input in your LSTM. If you have <span class="math-container">$C$</span> frames, the output of LSTM is
  <span class="math-container">$$
  Output, (h, c) = LSTM (frame \ features)
  $$</span>
  where frame features are size <span class="math-container">$(1, C, v)$</span>, where <span class="math-container">$v$</span> is the feature vector length from ConvNet. <span class="math-container">$Output$</span> is size <span class="math-container">$(1, C, w)$</span> and <span class="math-container">$h$</span> is <span class="math-container">$(1, 1, w)$</span> where <span class="math-container">$w$</span> is the size of the hidden layer in LSTM, and <span class="math-container">$h$</span> is the last hidden layer. Now, you can resize <span class="math-container">$Output$</span> to <span class="math-container">$(C, w)$</span> and make it an input in some linear layer that outputs a batch size <span class="math-container">$C$</span>. This is your predictions per frame.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-20T11:26:17.993Z
Answers.LastActivityDate: 2021-02-20T11:26:17.993Z
Answers.UserDisplayName: Alex
Body: <p>If I'm dealing with a sequence of images as the input (frame by frame), and I want to output a matrix at each timestamp, can the hidden state be a matrix?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T04:59:35.700Z
Comments.Text: Please add more context, so that it's easier for people to help you.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T10:26:35.937Z
Comments.Text: Hi Paul. Welcome to Artificial Intelligence Stack Exchange! As someone suggested above, could you please explain **why** are you asking whether the hidden state of an RNN can be a matrix? Why do you need it to be a matrix? This doesn't  seem to necessarily follow from what you wrote in your post so far.
Comments.UserDisplayName: nbro
Tags: <recurrent-neural-networks>
Title: Can the hidden state of an RNN be a matrix?
UserDisplayName: paul
ThreadText: # Can the hidden state of an RNN be a matrix?
  
  (Asked by: username_0 on 2021-02-20)
  
  If I'm dealing with a sequence of images as the input (frame by frame), and I want to output a matrix at each timestamp, can the hidden state be a matrix?
  
  **Tags:** recurrent-neural-networks
  > **Comments:**
  > **username_1**: Please add more context, so that it's easier for people to help you.  
  > **username_2**: Hi Paul. Welcome to Artificial Intelligence Stack Exchange! As someone suggested above, could you please explain **why** are you asking whether the hidden state of an RNN can be a matrix? Why do you need it to be a matrix? This doesn't  seem to necessarily follow from what you wrote in your post so far.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2021-02-20. Score: 3
  
  Yes, I would say more, that hidden state can be a tensor of arbitrary dimensionality. For vanilla RNN the update rules of the hidden state and the output are:
  $$
  h\_t = \sigma\_h(W\_h x\_t + U\_h h\_{t-1} + b\_h)
  $$
  $$
  y\_t = \sigma\_y(W\_y h\_t + b\_y)
  $$
  Here $W\_h$ is input to hidden state matrix, and $U\_h$ is hidden state to hidden state, $W\_y$ is hidden state to output. One can simply upgrade these matrix products to more general case of tensor products, such that they could handle multidimensional input data and hidden state.
  
  The contraction of the $m$-dimensional tensor $X$ with $n$-dimensional Y (we assume $m \geq n$) by maximal amount of indices will be $m-n$ dimensional tensor $Z$:
  $$
  X\_{i\_1 \ldots i\_m} Y\_{i\_1 \ldots i\_n} = Z\_{i\_{n+1} \ldots i\_m}
  $$
  
  Namely, imagine, that the $x\_t = (x\_t)\_{i\_1 \ldots i\_X}$ is an $d\_x$-dimensional tensor. Then take $W\_h$ to be $d\_{wh}$ dimensional tensor, hidden state $h$ to be $d\_h = d\_{wh} - d\_{x}$ -dimensional, $U\_h$ is $d\_{uh}$ - dimensional, $W\_y$ is $d\_{wy}$ -dimensional. The output $y\_t$ of the RNN will be $d\_{wy} - d\_{h}$ dimensional tensor.
  
  Here note, that in order for the sum to be make sense in the first equation, dimensions need to match:
  $$
  d\_{uh} - d\_{h} = d\_{h}
  $$
  
  
  ---
  
  ### Answer by username_4 on 2021-02-20. Score: 1
  
  The question is, why you specifically want a matrix? I assume you mean per-frame features. In such case you can use a ConvNet as a feature extractor, i.e. it outputs a vector of features of fixed size. This vector is an input in your LSTM. If you have $C$ frames, the output of LSTM is
  $$
  Output, (h, c) = LSTM (frame \ features)
  $$
  where frame features are size $(1, C, v)$, where $v$ is the feature vector length from ConvNet. $Output$ is size $(1, C, w)$ and $h$ is $(1, 1, w)$ where $w$ is the size of the hidden layer in LSTM, and $h$ is the last hidden layer. Now, you can resize $Output$ to $(C, w)$ and make it an input in some linear layer that outputs a batch size $C$. This is your predictions per frame.
  
  
  ---
  
  