Answers.Body: <p>For a standard convolution layer, the weight matrix will have a shape of <code>(out_channels, in_channels, kernel_sizes)</code>. In addition, you will need a vector of shape <code>[out_channels]</code> for biases. For your specific case, 2d, your weight matrix will have a shape of <code>(out_channels, in_channels, kernel_size[0], kernel_size[1])</code>.</p>
  <p>Now, if we plugin the numbers:</p>
  <ul>
  <li><code>out_channels = 10</code>, you're having 10 filters</li>
  <li><code>in_channels = 3</code>, the picture is RGB in this case, so there are 3 channels (the last dimension of the input)</li>
  <li><code>kernel_size[0] = kernel_size[1] = 3</code></li>
  </ul>
  <p>In total you're gonna have <code>10*3*3*3 + 10 = 280</code> parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-16T14:29:25.923Z
Answers.LastActivityDate: 2021-12-18T12:27:20.857Z
Answers.LastEditDate: 2021-12-18T12:27:20.857Z
Answers.UserDisplayName: razvanc92
Answers.Body: <h2>What are the parameters in a convolutional layer?</h2>
  
  <p>The (learnable) parameters of a convolutional layer are the elements of the kernels (or filters) and biases (if you decide to have them). There are 1d, 2d and 3d convolutions. The most common are 2d convolutions, which are the ones people usually refer to, so I will mainly focus on this case.</p>
  
  <h2>2d convolutions</h2>
  
  <h3>Example</h3>
  
  <p>If the 2d convolutional layer has <span class="math-container">$10$</span> filters of <span class="math-container">$3 \times 3$</span> shape and the input to the convolutional layer is <span class="math-container">$24 \times 24 \times 3$</span>, then this actually means that the filters will have shape <span class="math-container">$3 \times 3 \times 3$</span>, i.e. each filter will have the 3rd dimension that is equal to the 3rd dimension of the input. So, the 3rd dimension of the kernel is not given because it can be determined from the 3rd dimension of the input. </p>
  
  <p>2d convolutions are performed along only 2 axes (x and y), hence the name. Here's a picture of a typical 2d convolutional layer where the depth of the kernel (in orange) is equal to the depth of the input volume (in cyan).</p>
  
  <p><a href="https://i.sstatic.net/3folL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/3folL.png" alt="enter image description here"></a></p>
  
  <p>Each kernel can optionally have an associated scalar bias. </p>
  
  <p>At this point, you should already be able to calculate the number of parameters of a standard convolutional layer.  In your case, the number of parameters is <span class="math-container">$10 * (3*3*3) + 10 = 280$</span>.</p>
  
  <h3>A TensorFlow proof</h3>
  
  <p>The following simple TensorFlow (version 2) program can confirm this.</p>
  
  <pre><code>import tensorflow as tf
  
  
  def get_model(input_shape, num_classes=10):
      model = tf.keras.Sequential()
      model.add(tf.keras.layers.Input(shape=input_shape))
      model.add(tf.keras.layers.Conv2D(10, kernel_size=3, use_bias=True))
      model.add(tf.keras.layers.Flatten())
      model.add(tf.keras.layers.Dense(num_classes))
  
      model.summary()
  
      return model
  
  
  if __name__ == '__main__':
      input_shape = (24, 24, 3)
      get_model(input_shape)
  </code></pre>
  
  <p>You should try setting <code>use_bias</code> to <code>False</code> to understand how the number of parameters changes.</p>
  
  <h3>General case</h3>
  
  <p>So, in general, given <span class="math-container">$M$</span> filters of shape <span class="math-container">$K \times K$</span> and an input of shape <span class="math-container">$H \times W \times D$</span>, then the number of parameters of the standard 2d convolutional layer, with scalar biases, is <span class="math-container">$M * (K * K * D) + M$</span> and, without biases, is <span class="math-container">$M * (K * K * D)$</span>.</p>
  
  <p>See also these related questions <a href="https://ai.stackexchange.com/q/17783/2444">How is the depth of filters of hidden layers determined?</a> and <a href="https://ai.stackexchange.com/q/5769/2444">In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?</a>. </p>
  
  <h2>1d and 3d convolutions</h2>
  
  <p>There are also 1d and 3d convolutions. </p>
  
  <p>For example, in the case of 3d convolutions, the kernels may not have the same dimension as the depth of the input, so the number of parameters is calculated differently for 3d convolutional layers. Here's a diagram of 3d convolutional layer, where the kernel has a depth different than the depth of the input volume.</p>
  
  <p><a href="https://i.sstatic.net/F282Z.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/F282Z.png" alt="enter image description here"></a></p>
  
  <p>See e.g. <a href="https://stackoverflow.com/q/42883547/3924118">Intuitive understanding of 1D, 2D, and 3D convolutions in convolutional neural networks</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-17T00:36:54.023Z
Answers.Text: The diagrams were taken from https://ai.stackexchange.com/a/13786/2444.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-16T14:29:14.690Z
Answers.LastActivityDate: 2020-03-17T01:01:29.327Z
Answers.LastEditDate: 2020-03-17T01:01:29.327Z
Answers.UserDisplayName: nbro
Body: <p>I was recently asked at an interview to calculate the number of parameters for a convolutional layer. I am deeply ashamed to admit I didn't know how to do that, even though I've been working and using CNN for years now.</p>
  
  <p>Given a convolutional layer with ten <span class="math-container">$3 \times 3$</span> filters and an input of shape <span class="math-container">$24 \times 24 \times 3$</span>, what is the total number of parameters of this convolutional layer?</p>
  
Tags: <convolutional-neural-networks><filters><convolutional-layers><convolution-arithmetic>
Title: How to calculate the number of parameters of a convolutional layer?
UserDisplayName: Ælex
ThreadText: # How to calculate the number of parameters of a convolutional layer?
  
  (Asked by: username_0 on 2020-03-16)
  
  I was recently asked at an interview to calculate the number of parameters for a convolutional layer. I am deeply ashamed to admit I didn't know how to do that, even though I've been working and using CNN for years now.
  
  Given a convolutional layer with ten $3 \times 3$ filters and an input of shape $24 \times 24 \times 3$, what is the total number of parameters of this convolutional layer?
  
  **Tags:** convolutional-neural-networks, filters, convolutional-layers, convolution-arithmetic
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2020-03-16. Score: 7 | Accepted answer
  
  What are the parameters in a convolutional layer?
  The (learnable) parameters of a convolutional layer are the elements of the kernels (or filters) and biases (if you decide to have them). There are 1d, 2d and 3d convolutions. The most common are 2d convolutions, which are the ones people usually refer to, so I will mainly focus on this case.
  2d convolutions
  Example
  If the 2d convolutional layer has $10$ filters of $3 \times 3$ shape and the input to the convolutional layer is $24 \times 24 \times 3$, then this actually means that the filters will have shape $3 \times 3 \times 3$, i.e. each filter will have the 3rd dimension that is equal to the 3rd dimension of the input. So, the 3rd dimension of the kernel is not given because it can be determined from the 3rd dimension of the input. 
  2d convolutions are performed along only 2 axes (x and y), hence the name. Here's a picture of a typical 2d convolutional layer where the depth of the kernel (in orange) is equal to the depth of the input volume (in cyan).
  
  Each kernel can optionally have an associated scalar bias. 
  At this point, you should already be able to calculate the number of parameters of a standard convolutional layer.  In your case, the number of parameters is $10 * (3*3*3) + 10 = 280$.
  A TensorFlow proof
  The following simple TensorFlow (version 2) program can confirm this.
  import tensorflow as tf
  
  
  def get_model(input_shape, num_classes=10):
      model = tf.keras.Sequential()
      model.add(tf.keras.layers.Input(shape=input_shape))
      model.add(tf.keras.layers.Conv2D(10, kernel_size=3, use_bias=True))
      model.add(tf.keras.layers.Flatten())
      model.add(tf.keras.layers.Dense(num_classes))
  
      model.summary()
  
      return model
  
  
  if __name__ == '__main__':
      input_shape = (24, 24, 3)
      get_model(input_shape)
  
  You should try setting use_bias to False to understand how the number of parameters changes.
  General case
  So, in general, given $M$ filters of shape $K \times K$ and an input of shape $H \times W \times D$, then the number of parameters of the standard 2d convolutional layer, with scalar biases, is $M * (K * K * D) + M$ and, without biases, is $M * (K * K * D)$.
  See also these related questions How is the depth of filters of hidden layers determined? and In a CNN, does each new filter have different weights for each input channel, or are the same weights of each filter used across input channels?. 
  1d and 3d convolutions
  There are also 1d and 3d convolutions. 
  For example, in the case of 3d convolutions, the kernels may not have the same dimension as the depth of the input, so the number of parameters is calculated differently for 3d convolutional layers. Here's a diagram of 3d convolutional layer, where the kernel has a depth different than the depth of the input volume.
  
  See e.g. Intuitive understanding of 1D, 2D, and 3D convolutions in convolutional neural networks.
  
  > **Comments:**
  > **username_2**: The diagrams were taken from https://ai.stackexchange.com/a/13786/2444.  
  
  ---
  
  ### Answer by username_1 on 2020-03-16. Score: 3
  
  For a standard convolution layer, the weight matrix will have a shape of `(out_channels, in_channels, kernel_sizes)`. In addition, you will need a vector of shape `[out_channels]` for biases. For your specific case, 2d, your weight matrix will have a shape of `(out_channels, in_channels, kernel_size[0], kernel_size[1])`.
  
  Now, if we plugin the numbers:
  
  * `out_channels = 10`, you're having 10 filters
  * `in_channels = 3`, the picture is RGB in this case, so there are 3 channels (the last dimension of the input)
  * `kernel_size[0] = kernel_size[1] = 3`
  
  In total you're gonna have `10*3*3*3 + 10 = 280` parameters.
  
  
  ---
  
  