Answers.Body: <p>A dense layer is a function that maps an input <span class="math-container">$\mathbf{x} \in \mathbb{R}^{in}$</span> onto an output <span class="math-container">$\mathbf{y} \in \mathbb{R}^{out}$</span>. This is implemented as a matrix multiplication, and to make the above computation work, the matrix is an <span class="math-container">$in \times out$</span>-Matrix and has therefore <span class="math-container">$in \cdot out$</span> many parameters. If you use a dense layer with bias (like in your case), there will be a vector <span class="math-container">$\mathbf{b} \in \mathbb{R}^{out}$</span> added to the output, which increases the number of parameters of the layer to <span class="math-container">$in \times out + out$</span>.</p>
  <p>In your case: You have a 1-dimensional input and apply a layer with two output neurons, you get <span class="math-container">$1 \cdot 2 + 2 = 4$</span> parameters in the first layer. The subsequent layer (also 2 neurons) will have <span class="math-container">$2 \cdot 2 + 2 = 6$</span> parameters. And the last layer has <span class="math-container">$2 \cdot 1 + 1 = 3$</span> parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T17:35:53.523Z
Answers.LastActivityDate: 2024-01-08T17:35:53.523Z
Answers.UserDisplayName: Chillston
Body: <pre><code>from tensorflow import keras
  import matplotlib.pyplot as plt
  
  layers = keras.layers
  N_train = 10 ** 4  # number of training samples
  # Note: &quot;[:, np.newaxis]&quot; reshapes array to (N,1) as required by our DNN (we input one feature per sample)
  xtrain = np.random.uniform(-10, 10, N_train)[:, np.newaxis]
  ytrain = some_complicated_function(xtrain) + np.random.randn(xtrain.shape[0])  # train data includes some noise 
  
  # Each xtrain element is added with each and every randomizing element
  
  nb_nodes = 2
  nb_layers = 2
  activation = &quot;ReLU&quot; # insert your choice of activation funciton
  reg_strategy = &quot;L1&quot;# define your regularization strategy - use L1 and L2 regularization
  
  model = keras.models.Sequential(name=&quot;1Dfit&quot;)
  model.add(layers.Dense(nb_nodes, activation=activation, kernel_regularizer=reg_strategy, input_dim=xtrain.shape[1]))  # first layer
  
  # insert your code for for creating additional layers 
  
  
  model.add(layers.Dense(nb_nodes, activation=activation, kernel_regularizer=reg_strategy, input_dim=xtrain.shape[1]))  # second layer
  
  model.add(layers.Dense(1))  # final layer
  
  print(model.summary())
  </code></pre>
  <p>The output of this code is</p>
  <p>Model: &quot;1Dfit&quot;</p>
  <hr />
  <h1>Layer (type)                Output Shape              Param #</h1>
  <p>dense_5 (Dense)             (None, 2)                 4</p>
  <p>dense_6 (Dense)             (None, 2)                 6</p>
  <p>dense_7 (Dense)             (None, 1)                 3</p>
  <p>=================================================================
  Total params: 13 (52.00 Byte)
  Trainable params: 13 (52.00 Byte)
  Non-trainable params: 0 (0.00 Byte)</p>
  <hr />
  <p>The number of nodes is 2 in each layer but the parameters are different. What am I missing and what do the number of parameters depend on?</p>
  
Tags: <neural-networks><machine-learning><tensorflow><models><dense-layers>
Title: Why does the number of parameters differ in each layer when each layer is defined the exact same way
UserDisplayName: Mohd. Farhan Hassan
ThreadText: # Why does the number of parameters differ in each layer when each layer is defined the exact same way
  
  (Asked by: username_0 on 2024-01-08)
  
  ```
  from tensorflow import keras
  import matplotlib.pyplot as plt
  
  layers = keras.layers
  N_train = 10 ** 4  # number of training samples
  # Note: "[:, np.newaxis]" reshapes array to (N,1) as required by our DNN (we input one feature per sample)
  xtrain = np.random.uniform(-10, 10, N_train)[:, np.newaxis]
  ytrain = some_complicated_function(xtrain) + np.random.randn(xtrain.shape[0])  # train data includes some noise 
  
  # Each xtrain element is added with each and every randomizing element
  
  nb_nodes = 2
  nb_layers = 2
  activation = "ReLU" # insert your choice of activation funciton
  reg_strategy = "L1"# define your regularization strategy - use L1 and L2 regularization
  
  model = keras.models.Sequential(name="1Dfit")
  model.add(layers.Dense(nb_nodes, activation=activation, kernel_regularizer=reg_strategy, input_dim=xtrain.shape[1]))  # first layer
  
  # insert your code for for creating additional layers 
  
  model.add(layers.Dense(nb_nodes, activation=activation, kernel_regularizer=reg_strategy, input_dim=xtrain.shape[1]))  # second layer
  
  model.add(layers.Dense(1))  # final layer
  
  print(model.summary())
  
  ```
  
  The output of this code is
  
  Model: "1Dfit"
  
  ---
  
  # Layer (type) Output Shape Param #
  
  dense\_5 (Dense) (None, 2) 4
  
  dense\_6 (Dense) (None, 2) 6
  
  dense\_7 (Dense) (None, 1) 3
  
  =================================================================
  Total params: 13 (52.00 Byte)
  Trainable params: 13 (52.00 Byte)
  Non-trainable params: 0 (0.00 Byte)
  
  ---
  
  The number of nodes is 2 in each layer but the parameters are different. What am I missing and what do the number of parameters depend on?
  
  **Tags:** neural-networks, machine-learning, tensorflow, models, dense-layers
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-08. Score: 1
  
  A dense layer is a function that maps an input $\mathbf{x} \in \mathbb{R}^{in}$ onto an output $\mathbf{y} \in \mathbb{R}^{out}$. This is implemented as a matrix multiplication, and to make the above computation work, the matrix is an $in \times out$-Matrix and has therefore $in \cdot out$ many parameters. If you use a dense layer with bias (like in your case), there will be a vector $\mathbf{b} \in \mathbb{R}^{out}$ added to the output, which increases the number of parameters of the layer to $in \times out + out$.
  
  In your case: You have a 1-dimensional input and apply a layer with two output neurons, you get $1 \cdot 2 + 2 = 4$ parameters in the first layer. The subsequent layer (also 2 neurons) will have $2 \cdot 2 + 2 = 6$ parameters. And the last layer has $2 \cdot 1 + 1 = 3$ parameters.
  
  
  ---
  
  