Answers.Body: <p>Concerning parameter sharing.</p>
  <ol>
  <li>For the fully connected neural network you have an input of shape <code>(H_in * W_in * C_in)</code> and the output of shape <code>(H_out * W_out * C_out)</code>. This means, that each color of the pixel of the output feature map is connected to every color of the pixel from the input feature map. There is a separate learnable parameter for each pixel in the input image and the output.
  Hence, one gets this huge number of parameters :
  <code>(H_in * H_out *  W_in * W_out * C_in * C_out)</code></li>
  <li>In the convolutional layer the input is the image of shape <code>(H_in, W_in, C_in)</code> and the weights account for the neighborhood of the given pixel, say of size <code>K * K</code>. The output is obtained as a weighted sum of the given pixel and its neighborhood. There is a separate kernel for each pair of the input and output channel <code>(C_in, C_out)</code>, but the weights of the kernel (a tensor of shape <code>(K, K, C_in, C_out)</code> are independent of the location. Actually, this layer can accept images of any resolution, whereas, the fully connected can work only with a fixed resolution.
  Finally one has <code>(K, K, C_in, C_out)</code> parameters, which for the kernel size <code>K</code> much smaller, than the input resolution result into significant drop in the number of variables.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-21T09:39:48.423Z
Answers.Text: There is a separate kernel for each pair of the input and output channel (C_in, C_out), but the weights of the kernel (a tensor of shape (K, K, C_in, C_out) are independent of the location. Does this sentence refer to that one single kernel has the same weights and that kernel is convolved across the entire image?
Answers.UserDisplayName: Sidra
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-21T13:40:53.433Z
Answers.Text: @user15723449 precisely. For a given pair of input of channel, the same kernel of shape (K, K) is convolved across the entire image.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-19T08:42:56.140Z
Answers.LastActivityDate: 2021-06-19T08:42:56.140Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.Body: <p>The same features might be across any location in the image, for a face recognition model, if the data is not very good it might happen that along with learning the features such as eyes and lips it also learns their locations if we only apply the convolution operation at a specific locality. However if we apply a filter across the image it will learn the features and not just &quot;remember&quot; the locations. This concept is what is referred to as parameter sharing. This is not the case in full connected layers of neural networks as the weight map size is correlated to the input and output activations and we don't stack the outputs for each weight. Parameter sharing is an inherent phenomenon in CNNs</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-11T03:23:09.717Z
Answers.LastActivityDate: 2022-06-11T03:23:09.717Z
Answers.UserDisplayName: Amey Varhade
Body: <p>I am trying to understand the concept of parameter sharing in a convolution neural network from <a href="https://cs231n.github.io/convolutional-networks/#layers" rel="nofollow noreferrer">Parameter Sharing</a>. I have a few confusions:</p>
  <p>Parameter sharing refers to the fact that for generating a single activation map, we use the same kernel throughout the image. And for that activation map, the weights of that kernel remain the same through the image?</p>
  <p><strong>Denoting a single 2-dimensional slice of depth as a depth slice (e.g. a volume of size [55x55x96] has 96 depth slices, each of size [55x55]), we are going to constrain the neurons in each depth slice to use the same weights and bias.</strong></p>
  <p>Does the above paragraph refer to the fact that <em>output of neurons in one activation map is generated by using the same weights in kernel throughout the image? And that kernel is convolved on the entire image</em>?</p>
  <p><strong>No. of parameters without parameter sharing:</strong>
  There are 55x55x96 = 290,400 neurons in the first Conv Layer, and each has 11x11x3 = 363 weights and 1 bias. Together, this adds up to 290400 * 364 = 105,705,600 parameters on the first layer of the ConvNet alone. Clearly, this number is very high.</p>
  <p><strong>No. of parameters with parameter sharing</strong>
  With parameter sharing scheme, the first Conv Layer in our example would now have only 96 unique set of weights (one for each depth slice), for a total of 96x11x11x3 = 34,848 unique weights, or 34,944 parameters (+96 biases). <em><strong><em>Alternatively, all 55</em>55 neurons in each depth slice will now be using the same parameters.</strong></em>* What does this bold sentence mean?</p>
  <p>Also, how the parameters are different for both schemes? In both cases, we are using 96 kernels with 11x11x3 size and the resulting output is 55x55. Then how the number of parameters for both schemes coming out to be different?</p>
  
Tags: <convolutional-neural-networks>
Title: How is parameter sharing done in CNN?
UserDisplayName: Sidra
ThreadText: # How is parameter sharing done in CNN?
  
  (Asked by: username_0 on 2021-06-18)
  
  I am trying to understand the concept of parameter sharing in a convolution neural network from [Parameter Sharing](https://cs231n.github.io/convolutional-networks/#layers). I have a few confusions:
  
  Parameter sharing refers to the fact that for generating a single activation map, we use the same kernel throughout the image. And for that activation map, the weights of that kernel remain the same through the image?
  
  **Denoting a single 2-dimensional slice of depth as a depth slice (e.g. a volume of size [55x55x96] has 96 depth slices, each of size [55x55]), we are going to constrain the neurons in each depth slice to use the same weights and bias.**
  
  Does the above paragraph refer to the fact that *output of neurons in one activation map is generated by using the same weights in kernel throughout the image? And that kernel is convolved on the entire image*?
  
  **No. of parameters without parameter sharing:**
  There are 55x55x96 = 290,400 neurons in the first Conv Layer, and each has 11x11x3 = 363 weights and 1 bias. Together, this adds up to 290400 \* 364 = 105,705,600 parameters on the first layer of the ConvNet alone. Clearly, this number is very high.
  
  **No. of parameters with parameter sharing**
  With parameter sharing scheme, the first Conv Layer in our example would now have only 96 unique set of weights (one for each depth slice), for a total of 96x11x11x3 = 34,848 unique weights, or 34,944 parameters (+96 biases). ****Alternatively, all 55*55 neurons in each depth slice will now be using the same parameters.***\* What does this bold sentence mean?
  
  Also, how the parameters are different for both schemes? In both cases, we are using 96 kernels with 11x11x3 size and the resulting output is 55x55. Then how the number of parameters for both schemes coming out to be different?
  
  **Tags:** convolutional-neural-networks
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-06-19. Score: 2
  
  Concerning parameter sharing.
  
  1. For the fully connected neural network you have an input of shape `(H_in * W_in * C_in)` and the output of shape `(H_out * W_out * C_out)`. This means, that each color of the pixel of the output feature map is connected to every color of the pixel from the input feature map. There is a separate learnable parameter for each pixel in the input image and the output.
     Hence, one gets this huge number of parameters :
     `(H_in * H_out * W_in * W_out * C_in * C_out)`
  2. In the convolutional layer the input is the image of shape `(H_in, W_in, C_in)` and the weights account for the neighborhood of the given pixel, say of size `K * K`. The output is obtained as a weighted sum of the given pixel and its neighborhood. There is a separate kernel for each pair of the input and output channel `(C_in, C_out)`, but the weights of the kernel (a tensor of shape `(K, K, C_in, C_out)` are independent of the location. Actually, this layer can accept images of any resolution, whereas, the fully connected can work only with a fixed resolution.
     Finally one has `(K, K, C_in, C_out)` parameters, which for the kernel size `K` much smaller, than the input resolution result into significant drop in the number of variables.
  
  > **Comments:**
  > **username_0**: There is a separate kernel for each pair of the input and output channel (C_in, C_out), but the weights of the kernel (a tensor of shape (K, K, C_in, C_out) are independent of the location. Does this sentence refer to that one single kernel has the same weights and that kernel is convolved across the entire image?  
  > **username_1**: @user15723449 precisely. For a given pair of input of channel, the same kernel of shape (K, K) is convolved across the entire image.  
  
  ---
  
  ### Answer by username_2 on 2022-06-11. Score: 1
  
  The same features might be across any location in the image, for a face recognition model, if the data is not very good it might happen that along with learning the features such as eyes and lips it also learns their locations if we only apply the convolution operation at a specific locality. However if we apply a filter across the image it will learn the features and not just "remember" the locations. This concept is what is referred to as parameter sharing. This is not the case in full connected layers of neural networks as the weight map size is correlated to the input and output activations and we don't stack the outputs for each weight. Parameter sharing is an inherent phenomenon in CNNs
  
  
  ---
  
  