Answers.Body: <p>The normal CNN layer setup, is to include a learnable bias parameter per output channel. Bias is added to all feature &quot;pixels&quot; after the convolution is processed, and before the non-linear function is applied to the feature.</p>
  <p>For example the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D" rel="nofollow noreferrer">TensforFlow Keras Conv2D layer</a> has bias optional, but enabled by default.</p>
  <p>Making the bias a learnable &quot;kernel&quot; that adds separately would not do anything different from a single valued bias, unless the addition was applied to the new feature, after the convolution, and with a stride greater than 1. Because it would be applied to the output of the convolution, and have no mathematical relationship to it, the additive kernel could have different height, width, stride and padding settings to the convolution it was modifying.</p>
  <p>I think you could fix the stride to be equal to the height and width, because other than edges (when there was no padding) an additive kernel that overlapped itself would be equivalent to a smaller kernel that did not.</p>
  <p>Creating this idea as a custom layer in Torch or TensorFlow should be quite straightforward, so it is possible to give it a try, to see if it changes performance in any way. I don't really have a gut feeling on whether it would be useful, but most similar &quot;what if&quot; ideas for modifying neural networks turn out to have limited impact.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-16T19:24:17.780Z
Answers.LastActivityDate: 2023-08-17T08:08:24.630Z
Answers.LastEditDate: 2023-08-17T08:08:24.630Z
Answers.UserDisplayName: Neil Slater
Body: <p>We already know that the kernel slides around the image, multiplying the pixels with the parameters, so, what if additionally, we also have a kernel slide around the image and add values(different than the ones with multiply with)?</p>
  
Tags: <neural-networks><convolutional-neural-networks>
Title: Convolution layer, with biases too?
UserDisplayName: akzytr
ThreadText: # Convolution layer, with biases too?
  
  (Asked by: username_0 on 2023-08-16)
  
  We already know that the kernel slides around the image, multiplying the pixels with the parameters, so, what if additionally, we also have a kernel slide around the image and add values(different than the ones with multiply with)?
  
  **Tags:** neural-networks, convolutional-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-16. Score: 1
  
  The normal CNN layer setup, is to include a learnable bias parameter per output channel. Bias is added to all feature "pixels" after the convolution is processed, and before the non-linear function is applied to the feature.
  
  For example the [TensforFlow Keras Conv2D layer](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv2D) has bias optional, but enabled by default.
  
  Making the bias a learnable "kernel" that adds separately would not do anything different from a single valued bias, unless the addition was applied to the new feature, after the convolution, and with a stride greater than 1. Because it would be applied to the output of the convolution, and have no mathematical relationship to it, the additive kernel could have different height, width, stride and padding settings to the convolution it was modifying.
  
  I think you could fix the stride to be equal to the height and width, because other than edges (when there was no padding) an additive kernel that overlapped itself would be equivalent to a smaller kernel that did not.
  
  Creating this idea as a custom layer in Torch or TensorFlow should be quite straightforward, so it is possible to give it a try, to see if it changes performance in any way. I don't really have a gut feeling on whether it would be useful, but most similar "what if" ideas for modifying neural networks turn out to have limited impact.
  
  
  ---
  
  