Answers.Body: <p>I found out how to get 3D output from LSTMCell so that I can matmul with output weights + biases and subtract with expected values:</p>
  
  <ul>
  <li>Inputs &amp; expecteds should be: placeholder(,[times,batch_size,num_inp]) instead of batch_size first then times. However, tf.keras.layers.LSTM will ask for [batch_size,times,num_inp]</li>
  <li>Use tf.nn.static_rnn with a list of inputs, instead of 1 input</li>
  </ul>
  
  <p>Source code:</p>
  
  <pre><code>import tensorflow as tf;
  
  x = [[[1],[2],[3]],[[4],[5],[6]]];
  times = 2;
  bsize = 3;
  
  #3d input
  inputs = tf.placeholder(tf.float32, [times,bsize,1]);
  
  cell   = tf.nn.rnn_cell.BasicRNNCell(20);
  hids,_ = tf.nn.static_rnn(cell,tf.unstack(inputs,times),dtype=tf.float32);
  
  sess = tf.Session();
  init = tf.global_variables_initializer();
  sess.run(init);
  
  #results in 2d
  print(sess.run(hids, {inputs:x}));
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-19T08:49:04.100Z
Answers.LastActivityDate: 2019-09-20T10:33:20.557Z
Answers.LastEditDate: 2019-09-20T10:33:20.557Z
Answers.UserDisplayName: Dan D.
Body: <p>Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data:</p>
  
  <pre><code>X = [[0,0],[0,1],[1,0],[1,1]];
  Y = [[0],  [1],  [1],  [0]  ];
  </code></pre>
  
  <p>However, RNN seems strange, I don't get it how to match X to Y, input of RNN layer is 3D and output is 2D (rightclick to open in new tab): <a href="https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6" rel="nofollow noreferrer">https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6</a></p>
  
  <pre><code>import tensorflow as tf;
  
  x = [[[1],[2],[3]], [[4],[5],[6]]];
  bsize = 2;
  times = 3;
  
  #3d input
  input = tf.placeholder(tf.float32, [bsize,times,1]);
  
  cell  = tf.keras.layers.LSTMCell(20);
  rnn   = tf.keras.layers.RNN(cell);
  hid   = rnn(input);
  
  sess = tf.Session();
  init = tf.global_variables_initializer();
  sess.run(init);
  
  #results in 2d
  print(sess.run(hid, {input:x}));
  </code></pre>
  
  <p>The example data seen on <a href="https://www.tensorflow.org/tutorials/sequences/recurrent" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/sequences/recurrent</a> are:</p>
  
  <pre><code> t=0  t=1    t=2  t=3     t=4
  [the, brown, fox, is,     quick]
  [the, red,   fox, jumped, high]
  </code></pre>
  
  <p>How to map these data <strong>from X (3D</strong> input for RNN layer) <strong>to Y (2D</strong>)? (Y is 2D because RNN layer output is 2D).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-18T11:21:11.240Z
Comments.Text: I'm not 100% sure I underststood the question but I'll try to answer it. By default keras is returning a tensor of shape: (batch_size, state_size) which corresponds to the last hidden step for each batch. If you want to return all the hidden states you need to set return_sequences to True, and it will return a 3d tensor with shape: (batch_size, timesteps, output_size).
Comments.UserDisplayName: razvanc92
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-18T11:54:53.143Z
Comments.Text: @razvanc92 actually 2d or 3d is the side matter, the main thing i would like to ask is how to create the training data X, Y for the brown-fox-red-fox example data on  https://www.tensorflow.org/tutorials/sequences/recurrent
Comments.UserDisplayName: Dan D.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-18T12:15:57.333Z
Comments.Text: The very first step you need to do, is to embed each work into a fixed size vector. Afterwards from my understanding, at time t=0 the input will be X[:, 0] or [the, the] in your case. The target will be the next time stamp t=1 X[:,1] or [brown, red]. Following this logic you got X already, and for Y it will be X[:, 1:] (all the words from X, pushed by one plus probably a special character to mark the end of the sentence.).
Comments.UserDisplayName: razvanc92
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-19T02:25:13.910Z
Comments.Text: @razvanc92 i know about embedding, but i'm just trying out with class index first
Comments.UserDisplayName: Dan D.
Tags: <deep-learning><tensorflow><recurrent-neural-networks><long-short-term-memory><deep-neural-networks>
Title: How to map X to Y for TensorFlow RNN training data
UserDisplayName: Dan D.
ThreadText: # How to map X to Y for TensorFlow RNN training data
  
  (Asked by: username_0 on 2019-09-18)
  
  Usually for DNN, I have the training data of matching X (2D) to Y (2D), for example, XOR data:
  
  ```
  X = [[0,0],[0,1],[1,0],[1,1]];
  Y = [[0],  [1],  [1],  [0]  ];
  
  ```
  
  However, RNN seems strange, I don't get it how to match X to Y, input of RNN layer is 3D and output is 2D (rightclick to open in new tab): <https://colab.research.google.com/drive/17IgFuxOYgN5fNO9LKwDijEBkIeWNPas6>
  
  ```
  import tensorflow as tf;
  
  x = [[[1],[2],[3]], [[4],[5],[6]]];
  bsize = 2;
  times = 3;
  
  #3d input
  input = tf.placeholder(tf.float32, [bsize,times,1]);
  
  cell  = tf.keras.layers.LSTMCell(20);
  rnn   = tf.keras.layers.RNN(cell);
  hid   = rnn(input);
  
  sess = tf.Session();
  init = tf.global_variables_initializer();
  sess.run(init);
  
  #results in 2d
  print(sess.run(hid, {input:x}));
  
  ```
  
  The example data seen on <https://www.tensorflow.org/tutorials/sequences/recurrent> are:
  
  ```
   t=0  t=1    t=2  t=3     t=4
  [the, brown, fox, is,     quick]
  [the, red,   fox, jumped, high]
  
  ```
  
  How to map these data **from X (3D** input for RNN layer) **to Y (2D**)? (Y is 2D because RNN layer output is 2D).
  
  **Tags:** deep-learning, tensorflow, recurrent-neural-networks, long-short-term-memory, deep-neural-networks
  
  > **Comments:**
  > **username_1**: I'm not 100% sure I underststood the question but I'll try to answer it. By default keras is returning a tensor of shape: (batch_size, state_size) which corresponds to the last hidden step for each batch. If you want to return all the hidden states you need to set return_sequences to True, and it will return a 3d tensor with shape: (batch_size, timesteps, output_size).  
  > **username_0**: @username_1 actually 2d or 3d is the side matter, the main thing i would like to ask is how to create the training data X, Y for the brown-fox-red-fox example data on  https://www.tensorflow.org/tutorials/sequences/recurrent  
  > **username_1**: The very first step you need to do, is to embed each work into a fixed size vector. Afterwards from my understanding, at time t=0 the input will be X[:, 0] or [the, the] in your case. The target will be the next time stamp t=1 X[:,1] or [brown, red]. Following this logic you got X already, and for Y it will be X[:, 1:] (all the words from X, pushed by one plus probably a special character to mark the end of the sentence.).  
  > **username_0**: @username_1 i know about embedding, but i'm just trying out with class index first  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-09-19. Score: 0 | Accepted answer
  
  I found out how to get 3D output from LSTMCell so that I can matmul with output weights + biases and subtract with expected values:
  
  * Inputs & expecteds should be: placeholder(,[times,batch\_size,num\_inp]) instead of batch\_size first then times. However, tf.keras.layers.LSTM will ask for [batch\_size,times,num\_inp]
  * Use tf.nn.static\_rnn with a list of inputs, instead of 1 input
  
  Source code:
  
  ```
  import tensorflow as tf;
  
  x = [[[1],[2],[3]],[[4],[5],[6]]];
  times = 2;
  bsize = 3;
  
  #3d input
  inputs = tf.placeholder(tf.float32, [times,bsize,1]);
  
  cell   = tf.nn.rnn_cell.BasicRNNCell(20);
  hids,_ = tf.nn.static_rnn(cell,tf.unstack(inputs,times),dtype=tf.float32);
  
  sess = tf.Session();
  init = tf.global_variables_initializer();
  sess.run(init);
  
  #results in 2d
  print(sess.run(hids, {inputs:x}));
  
  ```
  
  
  ---
  
  