Answers.Body: <ol>
  <li><p><strong>Is figure = code?</strong></p>
  
  <p>No.
  Your figure shows a fully connected feed forward network (<a href="https://en.wikipedia.org/wiki/Multilayer_perceptron" rel="nofollow noreferrer">MLP</a>). But in your code you are using a two layer <a href="https://en.wikipedia.org/wiki/Long_short-term_memory#Peephole_LSTM" rel="nofollow noreferrer">LSTM with peepholes</a>. For the visualization of LSTMs, blocks are usually used for each layer.</p>
  
  <p>Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: <a href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf" rel="nofollow noreferrer">Paper, fig. 1</a>).</p></li>
  </ol>
  
  <p><a href="https://i.sstatic.net/I0BPu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/I0BPu.png" alt="enter image description here"></a></p>
  
  <ol start="2">
  <li><p><strong>Why size 86?</strong></p>
  
  <p>The input is concatenated with the hidden state:
  <code>n_inputs + n_neurons = 36 + 50 = 86</code>.</p></li>
  <li><p><strong>Why size 100 in the second layer?</strong></p>
  
  <p>The second LSTM layer gets input of size 50 by the first layer (<code>n_neurons</code>) which is concatenated with the hidden state of the second layer (of size <code>n_neurons = 50</code>).
  Therefore you get <code>50 + 50 = 100</code>.</p></li>
  <li><p><strong>Why size 200?</strong></p>
  
  <p>There are four weight matrices of size <span class="math-container">$86 \times 50$</span> (fig.: colored circles and the g circle), which seem to be combined to one matrix (<span class="math-container">$4 \cdot 50$</span>) of size <span class="math-container">$86 \times 200$</span> (<code>layer0/kernel</code>).</p></li>
  <li><p><strong>Why size 50?</strong></p>
  
  <p>The three variables <code>w_f_diag</code>, <code>w_i_diag</code> and <code>w_o_diag</code> are for the peephole connections (fig: dashed lines) and they have the size of <code>n_neurons = 50</code>.</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-29T08:11:01.800Z
Answers.Text: Just one more query. Is the model I am using back propagation and forward propagation? or is it just forward propagation?
Answers.UserDisplayName: Jaffer Wilson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-29T08:27:25.567Z
Answers.Text: The model uses backpropagation for weight adaption during the training process. It (LSTM) is not a feed forward neural network, but a recurrent neural network, as it has recurrent connections (blue arrows in the figure).
Answers.UserDisplayName: dexteritas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-28T17:56:52.160Z
Answers.LastActivityDate: 2019-01-28T18:23:25.827Z
Answers.LastEditDate: 2019-01-28T18:23:25.827Z
Answers.UserDisplayName: dexteritas
Body: <p>I have the following program for my neural network:  </p>
  
  <pre><code>n_steps = 9
  n_inputs = 36
  n_neurons = 50
  n_outputs = 1
  n_layers = 2
  learning_rate = 0.0001
  batch_size =100
  n_epochs = 1000#200 
  train_set_size = 1000
  test_set_size = 1000
  tf.reset_default_graph()
  X = tf.placeholder(tf.float32, [None, n_steps, n_inputs],name="input")
  y = tf.placeholder(tf.float32, [None, n_outputs],name="output")
  layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer))
           for layer in range(n_layers)]    layers.append(tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer)))
  multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
  rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
  stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons]) 
  stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
  outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
  outputs = outputs[:,n_steps-1,:]
  </code></pre>
  
  <p>I want to know whether my network is fully connected or not?<br>
  When I try to see the variables, I see:  </p>
  
  <pre><code>multi_layer_cell.weights
  </code></pre>
  
  <p>The output is:  </p>
  
  <pre><code>[&lt;tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/kernel:0' shape=(86, 200) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/bias:0' shape=(200,) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_f_diag:0' shape=(50,) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_i_diag:0' shape=(50,) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_0/layer0/w_o_diag:0' shape=(50,) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/kernel:0' shape=(100, 200) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/bias:0' shape=(200,) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_f_diag:0' shape=(50,) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_i_diag:0' shape=(50,) dtype=float32_ref&gt;,
   &lt;tf.Variable 'rnn/multi_rnn_cell/cell_1/layer1/w_o_diag:0' shape=(50,) dtype=float32_ref&gt;]
  </code></pre>
  
  <p>I didn't understood whether each layer is getting the complete inputs or not.<br>
  I want to know whether the following figure is correct for the above code:<br>
  <a href="https://i.sstatic.net/ub8Qb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ub8Qb.png" alt="enter image description here"></a> </p>
  
  <p>If this is not then what is the figure for the network? Please let me know.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-28T11:07:05.730Z
Comments.Text: Yes, it is a feed forward fully connected network.
Comments.UserDisplayName: Patel Sunil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-01-28T11:21:53.080Z
Comments.Text: @PatelSunil Can you please tell me why there is 86,200. Why 200? Why 86? I know 50 is the neurons per layer. Then why it starts with 100, 200? Why 100? I am a bit confused here. Please help me know.
Comments.UserDisplayName: Jaffer Wilson
Tags: <neural-networks><machine-learning><ai-design><long-short-term-memory>
Title: Is my Neural Network program fully connected?
UserDisplayName: Jaffer Wilson
ThreadText: # Is my Neural Network program fully connected?
  
  (Asked by: username_0 on 2019-01-28)
  
  I have the following program for my neural network:  
  n_steps = 9
  n_inputs = 36
  n_neurons = 50
  n_outputs = 1
  n_layers = 2
  learning_rate = 0.0001
  batch_size =100
  n_epochs = 1000#200 
  train_set_size = 1000
  test_set_size = 1000
  tf.reset_default_graph()
  X = tf.placeholder(tf.float32, [None, n_steps, n_inputs],name="input")
  y = tf.placeholder(tf.float32, [None, n_outputs],name="output")
  layers = [tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer))
           for layer in range(n_layers)]    layers.append(tf.contrib.rnn.LSTMCell(num_units=n_neurons,activation=tf.nn.relu6, use_peepholes = True,name="layer"+str(layer)))
  multi_layer_cell = tf.contrib.rnn.MultiRNNCell(layers)
  rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)
  stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons]) 
  stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)
  outputs = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])
  outputs = outputs[:,n_steps-1,:]
  
  I want to know whether my network is fully connected or not?
  When I try to see the variables, I see:  
  multi_layer_cell.weights
  
  The output is:  
  [,
   ,
   ,
   ,
   ,
   ,
   ,
   ,
   ,
   ]
  
  I didn't understood whether each layer is getting the complete inputs or not.
  I want to know whether the following figure is correct for the above code:
   
  If this is not then what is the figure for the network? Please let me know.
  
  **Tags:** neural-networks, machine-learning, ai-design, long-short-term-memory
  > **Comments:**
  > **username_1**: Yes, it is a feed forward fully connected network.  
  > **username_0**: @PatelSunil Can you please tell me why there is 86,200. Why 200? Why 86? I know 50 is the neurons per layer. Then why it starts with 100, 200? Why 100? I am a bit confused here. Please help me know.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-01-28. Score: 2 | Accepted answer
  
  1. **Is figure = code?**
  
     No.
     Your figure shows a fully connected feed forward network ([MLP](https://en.wikipedia.org/wiki/Multilayer_perceptron)). But in your code you are using a two layer [LSTM with peepholes](https://en.wikipedia.org/wiki/Long_short-term_memory#Peephole_LSTM). For the visualization of LSTMs, blocks are usually used for each layer.
  
     Here is a figure of the LSTM with peepholes which is the base of the tensorflow implementation (Source: [Paper, fig. 1](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43905.pdf)).
  
  [![enter image description here](https://i.sstatic.net/I0BPu.png)](https://i.sstatic.net/I0BPu.png)
  
  2. **Why size 86?**
  
     The input is concatenated with the hidden state:
     `n_inputs + n_neurons = 36 + 50 = 86`.
  3. **Why size 100 in the second layer?**
  
     The second LSTM layer gets input of size 50 by the first layer (`n_neurons`) which is concatenated with the hidden state of the second layer (of size `n_neurons = 50`).
     Therefore you get `50 + 50 = 100`.
  4. **Why size 200?**
  
     There are four weight matrices of size $86 \times 50$ (fig.: colored circles and the g circle), which seem to be combined to one matrix ($4 \cdot 50$) of size $86 \times 200$ (`layer0/kernel`).
  5. **Why size 50?**
  
     The three variables `w_f_diag`, `w_i_diag` and `w_o_diag` are for the peephole connections (fig: dashed lines) and they have the size of `n_neurons = 50`.
  
  > **Comments:**
  > **username_0**: Just one more query. Is the model I am using back propagation and forward propagation? or is it just forward propagation?  
  > **username_2**: The model uses backpropagation for weight adaption during the training process. It (LSTM) is not a feed forward neural network, but a recurrent neural network, as it has recurrent connections (blue arrows in the figure).  
  
  ---
  
  