Answers.Body: <p>If you're looking for filters with known effect, the Gaussian filters do smoothing, the Gabor filters are useful for edge detection, etc.</p>
  <p>Usually, in deep learning models where things are trained from scratch, the filters are randomly initialized and then learned by the model's training scheme. For the most part, without using any of the well-known kernels mentioned above.</p>
  <p>Clarification on for the most part: so filters aren't initialized with the goal of knowing exactly which feature it will activate on, but what will assist the training procedure. Recently, people have even trained ResNets with or without <em>batch normalization</em> by finding good initial points -- it's an ongoing field of research.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-24T18:00:11.893Z
Answers.LastActivityDate: 2020-12-11T14:42:40.140Z
Answers.LastEditDate: 2020-12-11T14:42:40.140Z
Answers.UserDisplayName: mshlis
Answers.Body: <h3>What are filters in image processing?</h3>
  <p>In the context of image processing (and, in general, signal processing), the <em>kernels</em> (also known as <em>filters</em>) are used to perform some specific operation on the image. For example, you can use a <a href="https://homepages.inf.ed.ac.uk/rbf/HIPR2/gsmooth.htm" rel="nofollow noreferrer">Gaussian filter</a> to smooth the image (including its edges).</p>
  <h3>What are filters in CNNs?</h3>
  <p>In the context of convolutional neural networks (CNNs), the filters (or kernels) are the <strong>learnable</strong> parameters of the model.</p>
  <p>Before training, the kernels are usually randomly initialised (so they are not usually hardcoded). During training, depending on the loss or error of the network (according to the loss function), the kernels (or filters) are updated, so that to minimise the loss (or error). After training, they are typically fixed. Incredibly, the filters learned by CNNs can be similar to the <a href="https://en.wikipedia.org/wiki/Gabor_filter" rel="nofollow noreferrer">Gabor filter</a> (which is thought to be related to our visual system [<a href="http://www.cns.nyu.edu/%7Etony/vns/readings/olshausen-field-1996.pdf" rel="nofollow noreferrer">1</a>]). See <a href="https://www.deeplearningbook.org/contents/convnets.html" rel="nofollow noreferrer">figure 9.19 of chapter 9 (p. 365) of the Deep Learning book by Goodfellow et al</a>.</p>
  <p>The number of kernels that are applied to a given input (and more than one kernel is often applied) in a CNN is a <a href="https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)" rel="nofollow noreferrer">hyper-parameter</a>.</p>
  <h3>What are the differences and similarities?</h3>
  <p>In both contexts, the words &quot;kernel&quot; and &quot;filter&quot; are roughly synonymous, so they are often used interchangeably. Furthermore, in both cases, the kernels are related to the <a href="https://ai.stackexchange.com/a/22000/2444">convolution (or cross-correlation) operation</a>. More specifically, the application of a filter, which is a function <span class="math-container">$h$</span>, to an input, which is another function <span class="math-container">$f$</span>, is equivalent to the convolution of <span class="math-container">$f$</span> and <span class="math-container">$h$</span>. In mathematics, this is often denoted by <span class="math-container">$f \circledast h = g$</span>, where <span class="math-container">$\circledast$</span> is the convolution operator and <span class="math-container">$g$</span> is the result of the convolution operation and is often called the <em>convolution</em> (of <span class="math-container">$f$</span> and <span class="math-container">$h$</span>). In the case of image processing, <span class="math-container">$g$</span> is the filtered image. In the case of CNNs, <span class="math-container">$g$</span> is often called an <a href="https://stats.stackexchange.com/a/292064/82135">activation map</a>.</p>
  <h3>Further reading</h3>
  <p>Take a look at <a href="https://ai.stackexchange.com/a/21967/2444">this</a> and <a href="https://ai.stackexchange.com/a/21401/2444">this</a> answers for more details about CNNs and the convolution operation, respectively.</p>
  <p>You may also want to have a look at <a href="http://cs231n.github.io/understanding-cnn/" rel="nofollow noreferrer">Visualizing what ConvNets learn
  </a> for some info about the visualization of the kernels learned during the training of a CNN.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-25T17:16:42.350Z
Answers.Text: Thanks for your answer @nbro . You are saying "kernels are the learnable parameters of the model". 
  My understanding was that we would also have a set of weights to back-prop, connecting the output of each layer to the input of the next one. Is this correct?
Answers.UserDisplayName: nvergontbij
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-25T20:14:23.320Z
Answers.Text: @nvergontbij In the case of CNNs, this is not the case, but this would be true in the case of feed-forward neural networks (or multi-layer perceptrons). In the case of CNNs, the kernels (or filters) are the weights of the network.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-24T20:39:44.520Z
Answers.LastActivityDate: 2020-12-11T15:13:30.177Z
Answers.LastEditDate: 2020-12-11T15:13:30.177Z
Answers.UserDisplayName: nbro
Body: <p>No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filters' weights are learned by means of back-prop.</p>
  <p>But how are kernels themselves selected? I know kernels had been used in image processing long before CNNs, hence I'd imagine there would be a set of filters based on kernels (see, for example, <a href="https://en.wikipedia.org/wiki/Kernel_(image_processing)#Details" rel="nofollow noreferrer">this article</a>) that are proven to be effective for edge detection and the likes.</p>
  <p>Reading around the web, I also found something about &quot;randomly generated kernels&quot;. Does anyone know if and when this practice is adopted?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-29T13:29:05.037Z
Comments.Text: [Here](https://ai.stackexchange.com/q/5092/2444) is a related (if not duplicate) question.
Comments.UserDisplayName: nbro
Tags: <convolutional-neural-networks><backpropagation><gradient-descent><filters>
Title: Are filters fixed or learned?
UserDisplayName: nvergontbij
ThreadText: # Are filters fixed or learned?
  
  (Asked by: username_0 on 2019-05-24)
  
  No matter what I google or what paper I read, I can't find an answer to my question. In a deep convolutional neural network, let's say AlexNet (Krizhevsky, 2012), filters' weights are learned by means of back-prop.
  
  But how are kernels themselves selected? I know kernels had been used in image processing long before CNNs, hence I'd imagine there would be a set of filters based on kernels (see, for example, [this article](https://en.wikipedia.org/wiki/Kernel_(image_processing)#Details)) that are proven to be effective for edge detection and the likes.
  
  Reading around the web, I also found something about "randomly generated kernels". Does anyone know if and when this practice is adopted?
  
  **Tags:** convolutional-neural-networks, backpropagation, gradient-descent, filters
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/5092/2444) is a related (if not duplicate) question.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-05-24. Score: 6 | Accepted answer
  
  ### What are filters in image processing?
  
  In the context of image processing (and, in general, signal processing), the *kernels* (also known as *filters*) are used to perform some specific operation on the image. For example, you can use a [Gaussian filter](https://homepages.inf.ed.ac.uk/rbf/HIPR2/gsmooth.htm) to smooth the image (including its edges).
  
  ### What are filters in CNNs?
  
  In the context of convolutional neural networks (CNNs), the filters (or kernels) are the **learnable** parameters of the model.
  
  Before training, the kernels are usually randomly initialised (so they are not usually hardcoded). During training, depending on the loss or error of the network (according to the loss function), the kernels (or filters) are updated, so that to minimise the loss (or error). After training, they are typically fixed. Incredibly, the filters learned by CNNs can be similar to the [Gabor filter](https://en.wikipedia.org/wiki/Gabor_filter) (which is thought to be related to our visual system [[1](http://www.cns.nyu.edu/%7Etony/vns/readings/olshausen-field-1996.pdf)]). See [figure 9.19 of chapter 9 (p. 365) of the Deep Learning book by Goodfellow et al](https://www.deeplearningbook.org/contents/convnets.html).
  
  The number of kernels that are applied to a given input (and more than one kernel is often applied) in a CNN is a [hyper-parameter](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)).
  
  ### What are the differences and similarities?
  
  In both contexts, the words "kernel" and "filter" are roughly synonymous, so they are often used interchangeably. Furthermore, in both cases, the kernels are related to the [convolution (or cross-correlation) operation](https://ai.stackexchange.com/a/22000/2444). More specifically, the application of a filter, which is a function $h$, to an input, which is another function $f$, is equivalent to the convolution of $f$ and $h$. In mathematics, this is often denoted by $f \circledast h = g$, where $\circledast$ is the convolution operator and $g$ is the result of the convolution operation and is often called the *convolution* (of $f$ and $h$). In the case of image processing, $g$ is the filtered image. In the case of CNNs, $g$ is often called an [activation map](https://stats.stackexchange.com/a/292064/82135).
  
  ### Further reading
  
  Take a look at [this](https://ai.stackexchange.com/a/21967/2444) and [this](https://ai.stackexchange.com/a/21401/2444) answers for more details about CNNs and the convolution operation, respectively.
  
  You may also want to have a look at [Visualizing what ConvNets learn](http://cs231n.github.io/understanding-cnn/) for some info about the visualization of the kernels learned during the training of a CNN.
  
  > **Comments:**
  > **username_0**: Thanks for your answer @username_1 . You are saying "kernels are the learnable parameters of the model".  My understanding was that we would also have a set of weights to back-prop, connecting the output of each layer to the input of the next one. Is this correct?  
  > **username_1**: @username_0 In the case of CNNs, this is not the case, but this would be true in the case of feed-forward neural networks (or multi-layer perceptrons). In the case of CNNs, the kernels (or filters) are the weights of the network.  
  
  ---
  
  ### Answer by username_2 on 2019-05-24. Score: 1
  
  If you're looking for filters with known effect, the Gaussian filters do smoothing, the Gabor filters are useful for edge detection, etc.
  
  Usually, in deep learning models where things are trained from scratch, the filters are randomly initialized and then learned by the model's training scheme. For the most part, without using any of the well-known kernels mentioned above.
  
  Clarification on for the most part: so filters aren't initialized with the goal of knowing exactly which feature it will activate on, but what will assist the training procedure. Recently, people have even trained ResNets with or without *batch normalization* by finding good initial points -- it's an ongoing field of research.
  
  
  ---
  
  