Answers.Body: <p>The convolution operation performed by most CNNs that you will find (on the web) assumes that the signals/functions are discrete and 2-dimensional (e.g. images can be viewed as 2-dimensional discrete signals), although this does not have to be the case. In fact, 1D and 3D convolutions are also implemented in several deep learning libraries (see <a href="https://keras.io/api/layers/convolution_layers/" rel="nofollow noreferrer">here</a> for an example).</p>
  <p>The equation in your post describes the convolution operation for 1-dimensional continuous signals with domain <span class="math-container">$\mathbb{R}$</span> (sometimes called the <em>time domain</em>, hence the usage of the letter <span class="math-container">$t$</span>), so we use an integral that goes from <span class="math-container">$-\infty$</span> to <span class="math-container">$\infty$</span>.</p>
  <p>An equation that better describes the common 2D convolution operation performed by CNNs is the following</p>
  <p><span class="math-container">$$S(i, j)=(K * I)(i, j)=\sum_{m=-M}^M \sum_{n=-N}^N I(i+m, j+n) K(m, n), \tag{1}\label{1}$$</span></p>
  <p>where</p>
  <ul>
  <li><span class="math-container">$S(i, j)$</span> is the convolution of the signals <span class="math-container">$I$</span> (e.g. an image) and <span class="math-container">$K$</span> (e.g. the kernel) evaluated at the coordinates (or points of the domain) <span class="math-container">$(i, j)$</span>, so <span class="math-container">$S$</span> is the function that results from the convolution of <span class="math-container">$I$</span> and <span class="math-container">$K$</span></li>
  <li><span class="math-container">$(K * I)(i, j)$</span> is just another notation for <span class="math-container">$S(i, j)$</span>, which emphasizes that the convolution is performed between <span class="math-container">$K$</span> and <span class="math-container">$I$</span>, and <span class="math-container">$*$</span> is the convolution operation</li>
  <li><span class="math-container">$K(m, n)$</span> is the value of the function <span class="math-container">$K$</span> (in the context of neural networks and image processing, this is often known as the <em>kernel</em>, hence the <span class="math-container">$K$</span>, or <em>filter</em>) evaluated at <span class="math-container">$(m, n)$</span></li>
  <li>similarly, <span class="math-container">$I(i+m, j+n)$</span> is the value of the function <span class="math-container">$I$</span> (e.g. the image) evaluated at <span class="math-container">$(i+m, j+n)$</span></li>
  </ul>
  <p>So, essentially, equation \ref{1} is telling us that the value of the convolution between <span class="math-container">$I$</span> and <span class="math-container">$K$</span>, denoted by <span class="math-container">$S$</span>, evaluated at <span class="math-container">$(i, j)$</span> is a sum of products, where the factors are <span class="math-container">$I(i+m, j+n)$</span> and <span class="math-container">$K(m, n)$</span>, where the values of <span class="math-container">$m$</span> and <span class="math-container">$n$</span> depend on <span class="math-container">$M$</span> and <span class="math-container">$N$</span>, which could also be equal, and typical values would be, for example, <span class="math-container">$1$</span> or <span class="math-container">$2$</span>, so equation \ref{1} could be written as follows</p>
  <p><span class="math-container">$$S(i, j)=(K * I)(i, j)=\sum_{m=-1}^1 \sum_{n=-1}^1 I(i+m, j+n) K(m, n), \tag{2}\label{2}$$</span></p>
  <p>If you expand the summations, you will get</p>
  <p><span class="math-container">$$S(i, j)= 
  I(i-1, j-1) K(-1, -1) + I(i-1, j) K(-1, 0) + \dots + I(i+1, j+1) K(1, 1)$$</span></p>
  <p>This is essentially a 2D <a href="https://en.wikipedia.org/wiki/Dot_product" rel="nofollow noreferrer">dot product</a> or, equivalently, an element-wise multiplication followed by the addition of those multiplications.</p>
  <p>Another important thing to note is that the convolution of <span class="math-container">$I$</span> and <span class="math-container">$K$</span> evaluated at <span class="math-container">$(i, j)$</span> uses the values of <span class="math-container">$I$</span> with respect to the points <span class="math-container">$(i, j)$</span>.</p>
  <p>Moreover, in a 2D convolutional layer of a CNN, we do not just compute <span class="math-container">$S(i, j)$</span>, but we compute <span class="math-container">$S(i, j)$</span> for all possible values of <span class="math-container">$i$</span> and <span class="math-container">$j$</span>, which are typically the coordinates of the image (in the case of the first layer).</p>
  <p>Finally, the equation \ref{2} actually describes what is known as the <em>cross-correlation operation</em>, which is exactly the same as the convolution operation, but we use <span class="math-container">$+$</span> instead of <span class="math-container">$-$</span> in <span class="math-container">$I(i+m, j+n)$</span>. In practice, in the context of CNNs, it doesn't really matter whether you use the <span class="math-container">$+$</span> and <span class="math-container">$-$</span>, given that the kernels <span class="math-container">$K$</span> are learned. However, the convolution operation and the cross-correlation would be different in the case the kernels are not symmetric, but identical in the case they are. You can read more about this topic <a href="https://ai.stackexchange.com/a/22000/2444">in this answer that I had written a while ago</a>.</p>
  <p>You should probably also read <a href="https://www.deeplearningbook.org/contents/convnets.html" rel="nofollow noreferrer">this chapter</a> that more thoroughly describes all these operations. There are also other answers that I had written in the past that could be useful, such as <a href="https://ai.stackexchange.com/a/23831/2444">this</a>, <a href="https://ai.stackexchange.com/q/12510/2444">this</a>, <a href="https://ai.stackexchange.com/a/21401/2444">this</a>, <a href="https://ai.stackexchange.com/a/21967/2444">this</a>, and/or <a href="https://ai.stackexchange.com/a/21874/2444">this</a>. By the time you have understood all these answers, you will also have understood the convolution operation and how it is related to CNNs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-17T16:50:58.580Z
Answers.LastActivityDate: 2021-07-17T17:05:06.443Z
Answers.LastEditDate: 2021-07-17T17:05:06.443Z
Answers.UserDisplayName: nbro
Body: <p>I've been reading up on the convolution operation and neural networks. I understand that the convolution operation is defined as:</p>
  <p><span class="math-container">$$(f * g)(t)=\int_{-\infty}^{\infty} f(\tau) g(t-\tau) d \tau$$</span></p>
  <p>The convolution operation has some properties, such as commutativity, associativity, etc.</p>
  <p>How is the convolution connected to neural networks? How do we use this operation in a CNN?</p>
  
Tags: <neural-networks><convolutional-neural-networks><convolution>
Title: How is the convolution operation connected to neural networks?
UserDisplayName: Dariyoush
ThreadText: # How is the convolution operation connected to neural networks?
  
  (Asked by: username_0 on 2021-07-17)
  
  I've been reading up on the convolution operation and neural networks. I understand that the convolution operation is defined as:
  
  $$(f \* g)(t)=\int\_{-\infty}^{\infty} f(\tau) g(t-\tau) d \tau$$
  
  The convolution operation has some properties, such as commutativity, associativity, etc.
  
  How is the convolution connected to neural networks? How do we use this operation in a CNN?
  
  **Tags:** neural-networks, convolutional-neural-networks, convolution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-07-17. Score: 3 | Accepted answer
  
  The convolution operation performed by most CNNs that you will find (on the web) assumes that the signals/functions are discrete and 2-dimensional (e.g. images can be viewed as 2-dimensional discrete signals), although this does not have to be the case. In fact, 1D and 3D convolutions are also implemented in several deep learning libraries (see [here](https://keras.io/api/layers/convolution_layers/) for an example).
  
  The equation in your post describes the convolution operation for 1-dimensional continuous signals with domain $\mathbb{R}$ (sometimes called the *time domain*, hence the usage of the letter $t$), so we use an integral that goes from $-\infty$ to $\infty$.
  
  An equation that better describes the common 2D convolution operation performed by CNNs is the following
  
  $$S(i, j)=(K \* I)(i, j)=\sum\_{m=-M}^M \sum\_{n=-N}^N I(i+m, j+n) K(m, n), \tag{1}\label{1}$$
  
  where
  
  * $S(i, j)$ is the convolution of the signals $I$ (e.g. an image) and $K$ (e.g. the kernel) evaluated at the coordinates (or points of the domain) $(i, j)$, so $S$ is the function that results from the convolution of $I$ and $K$
  * $(K \* I)(i, j)$ is just another notation for $S(i, j)$, which emphasizes that the convolution is performed between $K$ and $I$, and $\*$ is the convolution operation
  * $K(m, n)$ is the value of the function $K$ (in the context of neural networks and image processing, this is often known as the *kernel*, hence the $K$, or *filter*) evaluated at $(m, n)$
  * similarly, $I(i+m, j+n)$ is the value of the function $I$ (e.g. the image) evaluated at $(i+m, j+n)$
  
  So, essentially, equation \ref{1} is telling us that the value of the convolution between $I$ and $K$, denoted by $S$, evaluated at $(i, j)$ is a sum of products, where the factors are $I(i+m, j+n)$ and $K(m, n)$, where the values of $m$ and $n$ depend on $M$ and $N$, which could also be equal, and typical values would be, for example, $1$ or $2$, so equation \ref{1} could be written as follows
  
  $$S(i, j)=(K \* I)(i, j)=\sum\_{m=-1}^1 \sum\_{n=-1}^1 I(i+m, j+n) K(m, n), \tag{2}\label{2}$$
  
  If you expand the summations, you will get
  
  $$S(i, j)=
  I(i-1, j-1) K(-1, -1) + I(i-1, j) K(-1, 0) + \dots + I(i+1, j+1) K(1, 1)$$
  
  This is essentially a 2D [dot product](https://en.wikipedia.org/wiki/Dot_product) or, equivalently, an element-wise multiplication followed by the addition of those multiplications.
  
  Another important thing to note is that the convolution of $I$ and $K$ evaluated at $(i, j)$ uses the values of $I$ with respect to the points $(i, j)$.
  
  Moreover, in a 2D convolutional layer of a CNN, we do not just compute $S(i, j)$, but we compute $S(i, j)$ for all possible values of $i$ and $j$, which are typically the coordinates of the image (in the case of the first layer).
  
  Finally, the equation \ref{2} actually describes what is known as the *cross-correlation operation*, which is exactly the same as the convolution operation, but we use $+$ instead of $-$ in $I(i+m, j+n)$. In practice, in the context of CNNs, it doesn't really matter whether you use the $+$ and $-$, given that the kernels $K$ are learned. However, the convolution operation and the cross-correlation would be different in the case the kernels are not symmetric, but identical in the case they are. You can read more about this topic [in this answer that I had written a while ago](https://ai.stackexchange.com/a/22000/2444).
  
  You should probably also read [this chapter](https://www.deeplearningbook.org/contents/convnets.html) that more thoroughly describes all these operations. There are also other answers that I had written in the past that could be useful, such as [this](https://ai.stackexchange.com/a/23831/2444), [this](https://ai.stackexchange.com/q/12510/2444), [this](https://ai.stackexchange.com/a/21401/2444), [this](https://ai.stackexchange.com/a/21967/2444), and/or [this](https://ai.stackexchange.com/a/21874/2444). By the time you have understood all these answers, you will also have understood the convolution operation and how it is related to CNNs.
  
  
  ---
  
  