<p>I found out how to get 3D output from LSTMCell so that I can matmul with output weights + biases and subtract with expected values:</p>

<ul>
<li>Inputs &amp; expecteds should be: placeholder(,[times,batch_size,num_inp]) instead of batch_size first then times. However, tf.keras.layers.LSTM will ask for [batch_size,times,num_inp]</li>
<li>Use tf.nn.static_rnn with a list of inputs, instead of 1 input</li>
</ul>

<p>Source code:</p>

<pre><code>import tensorflow as tf;

x = [[[1],[2],[3]],[[4],[5],[6]]];
times = 2;
bsize = 3;

#3d input
inputs = tf.placeholder(tf.float32, [times,bsize,1]);

cell   = tf.nn.rnn_cell.BasicRNNCell(20);
hids,_ = tf.nn.static_rnn(cell,tf.unstack(inputs,times),dtype=tf.float32);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hids, {inputs:x}));
</code></pre>

CC BY-SA 4.0
2019-09-19T08:49:04.100Z
2019-09-20T10:33:20.557Z
2019-09-20T10:33:20.557Z
Dan D.
<p>Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data:</p>

<pre><code>X = [[0,0],[0,1],[1,0],[1,1]];
Y = [[0],  [1],  [1],  [0]  ];
</code></pre>

<p>However, RNN seems strange, I don't get it how to match X to Y, input of RNN layer is 3D and output is 2D (rightclick to open in new tab): <a href="https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6" rel="nofollow noreferrer">https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6</a></p>

<pre><code>import tensorflow as tf;

x = [[[1],[2],[3]], [[4],[5],[6]]];
bsize = 2;
times = 3;

#3d input
input = tf.placeholder(tf.float32, [bsize,times,1]);

cell  = tf.keras.layers.LSTMCell(20);
rnn   = tf.keras.layers.RNN(cell);
hid   = rnn(input);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hid, {input:x}));
</code></pre>

<p>The example data seen on <a href="https://www.tensorflow.org/tutorials/sequences/recurrent" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/sequences/recurrent</a> are:</p>

<pre><code> t=0  t=1    t=2  t=3     t=4
[the, brown, fox, is,     quick]
[the, red,   fox, jumped, high]
</code></pre>

<p>How to map these data <strong>from X (3D</strong> input for RNN layer) <strong>to Y (2D</strong>)? (Y is 2D because RNN layer output is 2D).</p>

CC BY-SA 4.0
2019-09-18T11:21:11.240Z
I'm not 100% sure I underststood the question but I'll try to answer it. By default keras is returning a tensor of shape: (batch_size, state_size) which corresponds to the last hidden step for each batch. If you want to return all the hidden states you need to set return_sequences to True, and it will return a 3d tensor with shape: (batch_size, timesteps, output_size).
razvanc92
CC BY-SA 4.0
2019-09-18T11:54:53.143Z
@razvanc92 actually 2d or 3d is the side matter, the main thing i would like to ask is how to create the training data X, Y for the brown-fox-red-fox example data on  https://www.tensorflow.org/tutorials/sequences/recurrent
Dan D.
CC BY-SA 4.0
2019-09-18T12:15:57.333Z
The very first step you need to do, is to embed each work into a fixed size vector. Afterwards from my understanding, at time t=0 the input will be X[:, 0] or [the, the] in your case. The target will be the next time stamp t=1 X[:,1] or [brown, red]. Following this logic you got X already, and for Y it will be X[:, 1:] (all the words from X, pushed by one plus probably a special character to mark the end of the sentence.).
razvanc92
CC BY-SA 4.0
2019-09-19T02:25:13.910Z
@razvanc92 i know about embedding, but i'm just trying out with class index first
Dan D.
<deep-learning><tensorflow><recurrent-neural-networks><long-short-term-memory><deep-neural-networks>
How to map X to Y for TensorFlow RNN training data
Dan D.
# How to map X to Y for TensorFlow RNN training data

(Asked by: username_0 on 2019-09-18)

Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data:
X = [[0,0],[0,1],[1,0],[1,1]];
Y = [[0],  [1],  [1],  [0]  ];

However, RNN seems strange, I don't get it how to match X to Y, input of RNN layer is 3D and output is 2D (rightclick to open in new tab): https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6
import tensorflow as tf;

x = [[[1],[2],[3]], [[4],[5],[6]]];
bsize = 2;
times = 3;

#3d input
input = tf.placeholder(tf.float32, [bsize,times,1]);

cell  = tf.keras.layers.LSTMCell(20);
rnn   = tf.keras.layers.RNN(cell);
hid   = rnn(input);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hid, {input:x}));

The example data seen on https://www.tensorflow.org/tutorials/sequences/recurrent are:
 t=0  t=1    t=2  t=3     t=4
[the, brown, fox, is,     quick]
[the, red,   fox, jumped, high]

How to map these data from X (3D input for RNN layer) to Y (2D)? (Y is 2D because RNN layer output is 2D).

**Tags:** deep-learning, tensorflow, recurrent-neural-networks, long-short-term-memory, deep-neural-networks

> **Comments:**
> **username_1**: I'm not 100% sure I underststood the question but I'll try to answer it. By default keras is returning a tensor of shape: (batch_size, state_size) which corresponds to the last hidden step for each batch. If you want to return all the hidden states you need to set return_sequences to True, and it will return a 3d tensor with shape: (batch_size, timesteps, output_size).  
> **username_0**: @username_1 actually 2d or 3d is the side matter, the main thing i would like to ask is how to create the training data X, Y for the brown-fox-red-fox example data on  https://www.tensorflow.org/tutorials/sequences/recurrent  
> **username_1**: The very first step you need to do, is to embed each work into a fixed size vector. Afterwards from my understanding, at time t=0 the input will be X[:, 0] or [the, the] in your case. The target will be the next time stamp t=1 X[:,1] or [brown, red]. Following this logic you got X already, and for Y it will be X[:, 1:] (all the words from X, pushed by one plus probably a special character to mark the end of the sentence.).  
> **username_0**: @username_1 i know about embedding, but i'm just trying out with class index first  

---

## 1 answers

### Answer by username_0 on 2019-09-19. Score: 0 | Accepted answer

I found out how to get 3D output from LSTMCell so that I can matmul with output weights + biases and subtract with expected values:

Inputs & expecteds should be: placeholder(,[times,batch_size,num_inp]) instead of batch_size first then times. However, tf.keras.layers.LSTM will ask for [batch_size,times,num_inp]
Use tf.nn.static_rnn with a list of inputs, instead of 1 input

Source code:
import tensorflow as tf;

x = [[[1],[2],[3]],[[4],[5],[6]]];
times = 2;
bsize = 3;

#3d input
inputs = tf.placeholder(tf.float32, [times,bsize,1]);

cell   = tf.nn.rnn_cell.BasicRNNCell(20);
hids,_ = tf.nn.static_rnn(cell,tf.unstack(inputs,times),dtype=tf.float32);

sess = tf.Session();
init = tf.global_variables_initializer();
sess.run(init);

#results in 2d
print(sess.run(hids, {inputs:x}));


---

