Answers.Body: <p>Just as a short and quick answer to build of nbros:</p>
  <p>The way CNNs are typically taught, they are taught using a correlation on the forward pass, rather than a convolution. In reality, Convolutional neural networks is a bit of a misleading name, but not entirely incorrect.</p>
  <p>CNNs do in fact use convolutions every time they are trained and run. If a <em>correlation</em> is used on the forward pass, a <em>convolution</em> is used on the backward pass. The opposite is true if a convolution is used on the forward pass (which is equally valid as using a correlation). I couldn't seem to find this information anywhere, so had to learn it myself the hard way.</p>
  <p>So to summarise, a typical CNN goes like this: Correlation forward, convolution backward.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-19T05:03:28.097Z
Answers.Text: I'd also like to say this a really good question and a pretty important one for anyone who wants to properly understand the inner workings of a CNN. It can be very confusing when you go to do an actual convolution and find it's not doing what you expect.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-19T08:25:09.990Z
Answers.Text: Maybe you should explain why if the forward pass performs a convolution (or cross-correlation) the backward pass performs a cross-correlation (and, respectively, convolution). It was also my intention (although I haven't yet done it) to provide links to libraries and explain how they implement CNNs (in particular, the convolutional layers). If you know how certain libraries implement the CNNs (e.g. TensorFlow, PyTorch, etc.), then feel free to add this info to your answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-19T10:30:18.647Z
Answers.Text: Also, I don't think that convolutional neural network is a misleading name. You could have called them "cross-correlation neural networks" (but that would be long and without any benefit) or, to be more precise, "convolutional or cross-correlation, down- or/and up-sampling, fully or not, with possible other layers, such as skip layers, neural networks", but that name would be quite ridiculous.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T01:46:45.950Z
Answers.Text: @nbro I still think it's slightly misleading in the sense that it's very easy to assume that a convolution is being performed when calculating an output, when in reality it's normally a correlation. Given that this is the core element that makes a CNN what it is, it can be a bit confusing. I know I personally found it confusing.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-20T01:49:01.420Z
Answers.Text: @nbro As for why for the relationship between correlation/convolution and how libraries implement CNNs, this I don't know. I figured out the correlation forward convolution backward by just mucking around and doing a few forward and backward passes by hand, and I haven't much looked into the implementation within libraries such as Tensorflow or Pytorch.
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-19T04:59:34.363Z
Answers.LastActivityDate: 2020-06-19T04:59:34.363Z
Answers.UserDisplayName: Recessive
Answers.Body: <h1>Short answer</h1>
  <p>Theoretically, convolutional neural networks (CNNs) can either perform the cross-correlation or convolution: it does not really matter whether they perform the cross-correlation or convolution because the kernels are learnable, so they can adapt to the cross-correlation or convolution given the data, although, in the typical diagrams, CNNs are shown to perform the cross-correlation because (in libraries like TensorFlow) they are typically <em>implemented</em> with cross-correlations (and cross-correlations are conceptually simpler than convolutions). Moreover, in general, the kernels can or not be symmetric (although they typically won't be symmetric). In the case they are symmetric, the cross-correlation is equal to the convolution.</p>
  <h1>Long answer</h1>
  <p>To understand the answer to this question, I will provide two examples that show the similarities and differences between the convolution and cross-correlation operations. I will focus on the convolution and cross-correlation applied to 1-dimensional discrete and finite signals (which is the simplest case to which these operations can be applied) because, essentially, CNNs process finite and discrete signals (although typically higher-dimensional ones, but this answer applies to higher-dimensional signals too). Moreover, in this answer, I will assume that you are at least familiar with how the convolution (or cross-correlation) in a CNN is performed, so that I do not have to explain these operations in detail (otherwise this answer would be even longer).</p>
  <h2>What is the convolution and cross-correlation?</h2>
  <p>Both the convolution and the cross-correlation operations are defined as the dot product between a small matrix and different parts of another typically bigger matrix (in the case of CNNs, it is an image or a feature map). Here's the usual illustration (of the cross-correlation, but the idea of the convolution is the same!).</p>
  <p><a href="https://i.sstatic.net/Jhb6U.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Jhb6U.gif" alt="enter image description here" /></a></p>
  <h2>Example 1</h2>
  <p>To be more concrete, let's suppose that we have the output of a function (or signal) <span class="math-container">$f$</span> grouped in a matrix <span class="math-container">$$f = [2, 1, 3, 5, 4] \in \mathbb{R}^{1 \times 5},$$</span> and the output of a kernel function also grouped in another matrix <span class="math-container">$$h=[1, -1] \in \mathbb{R}^{1 \times 2}.$$</span>  For simplicity, let's assume that we do not pad the input signal and we perform the convolution and cross-correlation with a stride of 1 (I assume that you are familiar with the concepts of padding and stride).</p>
  <h3>Convolution</h3>
  <p>Then the <strong>convolution</strong> of <span class="math-container">$f$</span> with <span class="math-container">$h$</span>, denoted as <span class="math-container">$f \circledast h = g_1$</span>, where <span class="math-container">$\circledast$</span> is the convolution operator, is computed as follows</p>
  <p><span class="math-container">\begin{align}
  f \circledast h = g_1
  &amp;=\\
  [(-1)*2 + 1*1, \\
  (-1)*1 + 1*3, \\
  (-1)*3 + 1*5, \\
  (-1)*5+1*4] 
  &amp;=\\ 
  [-2 + 1, -1 + 3, -3 + 5, -5 + 4]
  &amp;=\\
  [-1, 2, 2, -1] \in \mathbb{R}^{1 \times 4}
  \end{align}</span></p>
  <p>So, the convolution of <span class="math-container">$f$</span> with <span class="math-container">$h$</span> is computed as a series of element-wise multiplications between the horizontally flipped kernel <span class="math-container">$h$</span>, i.e. <span class="math-container">$[-1, 1]$</span>, and each <span class="math-container">$1 \times 2$</span> window of <span class="math-container">$f$</span>, each of which is followed by a summation (i.e. a dot product). This follows from the definition of convolution (which I will not report here).</p>
  <h3>Cross-correlation</h3>
  <p>Similarly, the <strong>cross-correlation</strong> of <span class="math-container">$f$</span> with <span class="math-container">$h$</span>, denoted as <span class="math-container">$f \otimes h = g_2$</span>, where <span class="math-container">$\otimes$</span> is the cross-correlation operator, is also defined as a dot product between <span class="math-container">$h$</span> and different parts of <span class="math-container">$f$</span>, but without flipping the elements of the kernel before applying the element-wise multiplications, that is</p>
  <p><span class="math-container">\begin{align}
  f \otimes h = g_2
  &amp;=\\
  [1*2 + (-1)*1, \\ 
  1*1 + (-1)*3, \\ 
  1*3 + (-1)*5, \\ 
  1*5 + (-1)*4] 
  &amp;=\\ 
  [2 - 1, 1 - 3, 3 - 5, 5 - 4]
  &amp;=\\
  [1, -2, -2, 1] \in \mathbb{R}^{1 \times 4}
  \end{align}</span></p>
  <h3>Notes</h3>
  <ol>
  <li><p>The only difference between the convolution and cross-correlation operations is that, in the first case, the kernel is flipped (along all spatial dimensions) before being applied.</p>
  </li>
  <li><p>In both cases, the result is a <span class="math-container">$1 \times 4$</span> vector. If we had convolved <span class="math-container">$f$</span> with a <span class="math-container">$1 \times 1$</span> vector, the result would have been a <span class="math-container">$1 \times 5$</span> vector. Recall that we assumed no padding (i.e. we don't add dummy elements to the left or right borders of <span class="math-container">$f$</span>) and stride 1 (i.e. we shift the kernel to the right one element at a time). Similarly, if we had convolved <span class="math-container">$f$</span> with a <span class="math-container">$1 \times 3$</span>, the result would have been a <span class="math-container">$1 \times 3$</span> vector (as you will see from the next example).</p>
  </li>
  <li><p>The results of the convolution and cross-correlation, <span class="math-container">$g_1$</span> and <span class="math-container">$g_2$</span>, are different. Specifically, one is the negated version of the other. So, the result of the convolution is generally different than the result of the cross-correlation, given the same signals and kernels (as you might have suspected).</p>
  </li>
  </ol>
  <h2>Example 2: symmetric kernel</h2>
  <p>Now, let's convolve <span class="math-container">$f$</span> with a <span class="math-container">$1 \times 3$</span> kernel that is symmetric around the middle element, <span class="math-container">$h_2 = [-1, 2, -1]$</span>. Let's first compute the convolution.</p>
  <p><span class="math-container">\begin{align}
  f \circledast h_2 = g_3
  &amp;=\\
  [(-1)*2 + (2)*1 + (-1)*3,\\ 
  (-1)*1 + (2)*3 + (-1)*5,\\ 
  (-1)*3 + (2)*5 + (-1)*4] 
  &amp;=\\ 
  [-2 + 2 + -3,\\ 
  -1 + 6 + -5,\\ 
  -3 + 10 + -4] 
  &amp;=\\
  [-3, 0, 3]
  \in \mathbb{R}^{1 \times 3}
  \end{align}</span></p>
  <p>Now, let's compute the cross-correlation</p>
  <p><span class="math-container">\begin{align}
  f \otimes h_2 = g_4
  &amp;=\\
  [(-1)*2 + (2)*1 + (-1)*3, \\ 
  (-1)*1 + (2)*3 + (-1)*5, \\ 
  (-1)*3 + (2)*5 + (-1)*4] 
  &amp;=\\
  [-3, 0, 3]
  \in \mathbb{R}^{1 \times 3}
  \end{align}</span></p>
  <p>Yes, that's right! In this case, the result of the convolution and the cross-correlation is the same. This is because the kernel is symmetric around the middle element. This result applies to any convolution or cross-correlation in any dimension. For example, the convolution of the 2d Gaussian kernel (a centric-symmetric kernel) and a 2d image is equal to the cross-correlation of the same signals.</p>
  <h2>CNNs have learnable kernels</h2>
  <p>In the case of CNNs, the kernels are the learnable parameters, so we do not know beforehand whether the kernels will be symmetric or not around their middle element. They won't probably be. In any case, CNNs can perform either the cross-correlation (i.e. no flip of the filter) or convolution: it does not really matter if they perform cross-correlation or convolution because the filter is learnable and can adapt to the data and tasks that you want to solve, although, in the visualizations and diagrams, CNNs are typically shown to perform the cross-correlation (but this does not have to be the case in practice).</p>
  <h2>Do libraries implement the convolution or correlation?</h2>
  <p>In practice, certain libraries provide functions to compute both convolution and cross-correlation. For example, NumPy provides both the functions <a href="https://numpy.org/doc/stable/reference/generated/numpy.convolve.html" rel="nofollow noreferrer"><code>convolve</code></a> and <a href="https://numpy.org/doc/stable/reference/generated/numpy.correlate.html" rel="nofollow noreferrer"><code>correlate</code></a> to compute both the convolution and cross-correlation, respectively. If you execute the following piece of code (Python 3.7), you will get results that are consistent with my explanations above.</p>
  <pre><code># Using Python 3.11.9 and NumPy 2.0.2
  
  import numpy as np
  
  f = np.array([2.0, 1.0, 3.0, 5.0, 4.0])
  
  h = np.array([1.0, -1.0])
  
  conv = np.convolve(f, h, mode=&quot;valid&quot;)
  cross = np.correlate(f, h, mode=&quot;valid&quot;)
  
  print(f&quot;Convolution of f = {f} and h = {h} = {conv}&quot;)
  print(f&quot;Correlation of f = {f} and h = {h} = {cross}&quot;)
  
  h2 = np.array([-1.0, 2.0, -1.0])
  
  conv = np.convolve(f, h2, mode=&quot;valid&quot;)
  cross = np.correlate(f, h2, mode=&quot;valid&quot;)
  
  print(f&quot;Convolution of f = {f} and h2 = {h} = {conv}&quot;)
  print(f&quot;Correlation of f = {f} and h2 = {h} = {cross}&quot;)
  </code></pre>
  <p>Result</p>
  <pre><code>Convolution of f = [2. 1. 3. 5. 4.] and h = [ 1. -1.] = [-1.  2.  2. -1.]
  Correlation of f = [2. 1. 3. 5. 4.] and h = [ 1. -1.] = [ 1. -2. -2.  1.]
  Convolution of f = [2. 1. 3. 5. 4.] and h2 = [ 1. -1.] = [-3.  0.  3.]
  Correlation of f = [2. 1. 3. 5. 4.] and h2 = [ 1. -1.] = [-3.  0.  3.]
  </code></pre>
  <p>However, NumPy is not really a library that provides <a href="https://numpy.org/doc/stable/reference/generated/numpy.correlate.html" rel="nofollow noreferrer">out-of-the-box</a> functionality to build CNNs.</p>
  <p>On the other hand, TensorFlow's and <a href="https://pytorch.org/docs/master/generated/torch.nn.Conv1d.html" rel="nofollow noreferrer">PyTorch's</a> functions to build the convolutional layers actually perform cross-correlations. As I said above, although it does not really matter whether CNNs perform the convolution or cross-correlation, this naming is misleading. Here's a proof that TensorFlow's <a href="https://www.tensorflow.org/api_docs/python/tf/nn/conv1d" rel="nofollow noreferrer"><code>tf.nn.conv1d</code></a> actually implements the cross-correlation.</p>
  <pre><code># Using Python 3.11.9 and TensorFlow 2.18.0
  
  import tensorflow as tf
  
  f = tf.constant([2.0, 1.0, 3.0, 5.0, 4.0], dtype=tf.float32)
  h = tf.constant([1.0, -1.0], dtype=tf.float32)
  
  # Reshaping the inputs because conv1d accepts only certain shapes.
  f = tf.reshape(f, [1, int(f.shape[0]), 1])
  h = tf.reshape(h, [int(h.shape[0]), 1, 1])
  
  c = tf.nn.conv1d(f, h, stride=1, padding=&quot;VALID&quot;)
  
  # Using tf.squeeze to improve readability.
  print(f&quot;conv1d of f = {tf.squeeze(f)} and h = {tf.squeeze(h)} = {tf.squeeze(c)}&quot;)
  </code></pre>
  <p>Result</p>
  <pre><code>conv1d of f = [2. 1. 3. 5. 4.] and h = [ 1. -1.] = [ 1. -2. -2.  1.]
  </code></pre>
  <h2>Further reading</h2>
  <p>After having written this answer, I found the article <a href="https://glassboxmedicine.com/2019/07/26/convolution-vs-cross-correlation" rel="nofollow noreferrer">Convolution vs. Cross-Correlation</a> (2019) by Rachel Draelos, which essentially says the same thing that I am saying here, but provides more details and examples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-18T17:13:00.543Z
Answers.LastActivityDate: 2024-12-25T15:07:21.447Z
Answers.LastEditDate: 2024-12-25T15:07:21.447Z
Answers.UserDisplayName: nbro
Body: <p>Typically, people say that convolutional neural networks (CNN) perform the convolution operation, hence their name. However, some people have also said that a CNN actually performs the cross-correlation operation rather than the convolution. How is that? Does a CNN perform the convolution or cross-correlation operation? What is the difference between the convolution and cross-correlation operations?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-12-26T06:12:47.187Z
Comments.Text: The difference between them is that [convolution "flips" the filter kernel, but cross-correlation does not](https://dsp.stackexchange.com/a/94212/55647). Mathematically, cross-correlation and convolution are [adjoints of each other](https://dsp.stackexchange.com/a/89575/55647)
Comments.UserDisplayName: Gillespie
Tags: <neural-networks><convolutional-neural-networks><comparison><convolution><cross-correlation>
Title: Do convolutional neural networks perform convolution or cross-correlation?
UserDisplayName: nbro
ThreadText: # Do convolutional neural networks perform convolution or cross-correlation?
  
  (Asked by: username_0 on 2020-06-18)
  
  Typically, people say that convolutional neural networks (CNN) perform the convolution operation, hence their name. However, some people have also said that a CNN actually performs the cross-correlation operation rather than the convolution. How is that? Does a CNN perform the convolution or cross-correlation operation? What is the difference between the convolution and cross-correlation operations?
  
  **Tags:** neural-networks, convolutional-neural-networks, comparison, convolution, cross-correlation
  
  > **Comments:**
  > **username_1**: The difference between them is that [convolution "flips" the filter kernel, but cross-correlation does not](https://dsp.stackexchange.com/a/94212/55647). Mathematically, cross-correlation and convolution are [adjoints of each other](https://dsp.stackexchange.com/a/89575/55647)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2020-06-18. Score: 10 | Accepted answer
  
  # Short answer
  
  Theoretically, convolutional neural networks (CNNs) can either perform the cross-correlation or convolution: it does not really matter whether they perform the cross-correlation or convolution because the kernels are learnable, so they can adapt to the cross-correlation or convolution given the data, although, in the typical diagrams, CNNs are shown to perform the cross-correlation because (in libraries like TensorFlow) they are typically *implemented* with cross-correlations (and cross-correlations are conceptually simpler than convolutions). Moreover, in general, the kernels can or not be symmetric (although they typically won't be symmetric). In the case they are symmetric, the cross-correlation is equal to the convolution.
  
  # Long answer
  
  To understand the answer to this question, I will provide two examples that show the similarities and differences between the convolution and cross-correlation operations. I will focus on the convolution and cross-correlation applied to 1-dimensional discrete and finite signals (which is the simplest case to which these operations can be applied) because, essentially, CNNs process finite and discrete signals (although typically higher-dimensional ones, but this answer applies to higher-dimensional signals too). Moreover, in this answer, I will assume that you are at least familiar with how the convolution (or cross-correlation) in a CNN is performed, so that I do not have to explain these operations in detail (otherwise this answer would be even longer).
  
  ## What is the convolution and cross-correlation?
  
  Both the convolution and the cross-correlation operations are defined as the dot product between a small matrix and different parts of another typically bigger matrix (in the case of CNNs, it is an image or a feature map). Here's the usual illustration (of the cross-correlation, but the idea of the convolution is the same!).
  
  [![enter image description here](https://i.sstatic.net/Jhb6U.gif)](https://i.sstatic.net/Jhb6U.gif)
  
  ## Example 1
  
  To be more concrete, let's suppose that we have the output of a function (or signal) $f$ grouped in a matrix $$f = [2, 1, 3, 5, 4] \in \mathbb{R}^{1 \times 5},$$ and the output of a kernel function also grouped in another matrix $$h=[1, -1] \in \mathbb{R}^{1 \times 2}.$$ For simplicity, let's assume that we do not pad the input signal and we perform the convolution and cross-correlation with a stride of 1 (I assume that you are familiar with the concepts of padding and stride).
  
  ### Convolution
  
  Then the **convolution** of $f$ with $h$, denoted as $f \circledast h = g\_1$, where $\circledast$ is the convolution operator, is computed as follows
  
  \begin{align}
  f \circledast h = g\_1
  &=\\
  [(-1)\*2 + 1\*1, \\
  (-1)\*1 + 1\*3, \\
  (-1)\*3 + 1\*5, \\
  (-1)\*5+1\*4]
  &=\\
  [-2 + 1, -1 + 3, -3 + 5, -5 + 4]
  &=\\
  [-1, 2, 2, -1] \in \mathbb{R}^{1 \times 4}
  \end{align}
  
  So, the convolution of $f$ with $h$ is computed as a series of element-wise multiplications between the horizontally flipped kernel $h$, i.e. $[-1, 1]$, and each $1 \times 2$ window of $f$, each of which is followed by a summation (i.e. a dot product). This follows from the definition of convolution (which I will not report here).
  
  ### Cross-correlation
  
  Similarly, the **cross-correlation** of $f$ with $h$, denoted as $f \otimes h = g\_2$, where $\otimes$ is the cross-correlation operator, is also defined as a dot product between $h$ and different parts of $f$, but without flipping the elements of the kernel before applying the element-wise multiplications, that is
  
  \begin{align}
  f \otimes h = g\_2
  &=\\
  [1\*2 + (-1)\*1, \\
  1\*1 + (-1)\*3, \\
  1\*3 + (-1)\*5, \\
  1\*5 + (-1)\*4]
  &=\\
  [2 - 1, 1 - 3, 3 - 5, 5 - 4]
  &=\\
  [1, -2, -2, 1] \in \mathbb{R}^{1 \times 4}
  \end{align}
  
  ### Notes
  
  1. The only difference between the convolution and cross-correlation operations is that, in the first case, the kernel is flipped (along all spatial dimensions) before being applied.
  2. In both cases, the result is a $1 \times 4$ vector. If we had convolved $f$ with a $1 \times 1$ vector, the result would have been a $1 \times 5$ vector. Recall that we assumed no padding (i.e. we don't add dummy elements to the left or right borders of $f$) and stride 1 (i.e. we shift the kernel to the right one element at a time). Similarly, if we had convolved $f$ with a $1 \times 3$, the result would have been a $1 \times 3$ vector (as you will see from the next example).
  3. The results of the convolution and cross-correlation, $g\_1$ and $g\_2$, are different. Specifically, one is the negated version of the other. So, the result of the convolution is generally different than the result of the cross-correlation, given the same signals and kernels (as you might have suspected).
  
  ## Example 2: symmetric kernel
  
  Now, let's convolve $f$ with a $1 \times 3$ kernel that is symmetric around the middle element, $h\_2 = [-1, 2, -1]$. Let's first compute the convolution.
  
  \begin{align}
  f \circledast h\_2 = g\_3
  &=\\
  [(-1)\*2 + (2)\*1 + (-1)\*3,\\
  (-1)\*1 + (2)\*3 + (-1)\*5,\\
  (-1)\*3 + (2)\*5 + (-1)\*4]
  &=\\
  [-2 + 2 + -3,\\
  -1 + 6 + -5,\\
  -3 + 10 + -4]
  &=\\
  [-3, 0, 3]
  \in \mathbb{R}^{1 \times 3}
  \end{align}
  
  Now, let's compute the cross-correlation
  
  \begin{align}
  f \otimes h\_2 = g\_4
  &=\\
  [(-1)\*2 + (2)\*1 + (-1)\*3, \\
  (-1)\*1 + (2)\*3 + (-1)\*5, \\
  (-1)\*3 + (2)\*5 + (-1)\*4]
  &=\\
  [-3, 0, 3]
  \in \mathbb{R}^{1 \times 3}
  \end{align}
  
  Yes, that's right! In this case, the result of the convolution and the cross-correlation is the same. This is because the kernel is symmetric around the middle element. This result applies to any convolution or cross-correlation in any dimension. For example, the convolution of the 2d Gaussian kernel (a centric-symmetric kernel) and a 2d image is equal to the cross-correlation of the same signals.
  
  ## CNNs have learnable kernels
  
  In the case of CNNs, the kernels are the learnable parameters, so we do not know beforehand whether the kernels will be symmetric or not around their middle element. They won't probably be. In any case, CNNs can perform either the cross-correlation (i.e. no flip of the filter) or convolution: it does not really matter if they perform cross-correlation or convolution because the filter is learnable and can adapt to the data and tasks that you want to solve, although, in the visualizations and diagrams, CNNs are typically shown to perform the cross-correlation (but this does not have to be the case in practice).
  
  ## Do libraries implement the convolution or correlation?
  
  In practice, certain libraries provide functions to compute both convolution and cross-correlation. For example, NumPy provides both the functions [`convolve`](https://numpy.org/doc/stable/reference/generated/numpy.convolve.html) and [`correlate`](https://numpy.org/doc/stable/reference/generated/numpy.correlate.html) to compute both the convolution and cross-correlation, respectively. If you execute the following piece of code (Python 3.7), you will get results that are consistent with my explanations above.
  
  ```
  # Using Python 3.11.9 and NumPy 2.0.2
  
  import numpy as np
  
  f = np.array([2.0, 1.0, 3.0, 5.0, 4.0])
  
  h = np.array([1.0, -1.0])
  
  conv = np.convolve(f, h, mode="valid")
  cross = np.correlate(f, h, mode="valid")
  
  print(f"Convolution of f = {f} and h = {h} = {conv}")
  print(f"Correlation of f = {f} and h = {h} = {cross}")
  
  h2 = np.array([-1.0, 2.0, -1.0])
  
  conv = np.convolve(f, h2, mode="valid")
  cross = np.correlate(f, h2, mode="valid")
  
  print(f"Convolution of f = {f} and h2 = {h} = {conv}")
  print(f"Correlation of f = {f} and h2 = {h} = {cross}")
  
  ```
  
  Result
  
  ```
  Convolution of f = [2. 1. 3. 5. 4.] and h = [ 1. -1.] = [-1.  2.  2. -1.]
  Correlation of f = [2. 1. 3. 5. 4.] and h = [ 1. -1.] = [ 1. -2. -2.  1.]
  Convolution of f = [2. 1. 3. 5. 4.] and h2 = [ 1. -1.] = [-3.  0.  3.]
  Correlation of f = [2. 1. 3. 5. 4.] and h2 = [ 1. -1.] = [-3.  0.  3.]
  
  ```
  
  However, NumPy is not really a library that provides [out-of-the-box](https://numpy.org/doc/stable/reference/generated/numpy.correlate.html) functionality to build CNNs.
  
  On the other hand, TensorFlow's and [PyTorch's](https://pytorch.org/docs/master/generated/torch.nn.Conv1d.html) functions to build the convolutional layers actually perform cross-correlations. As I said above, although it does not really matter whether CNNs perform the convolution or cross-correlation, this naming is misleading. Here's a proof that TensorFlow's [`tf.nn.conv1d`](https://www.tensorflow.org/api_docs/python/tf/nn/conv1d) actually implements the cross-correlation.
  
  ```
  # Using Python 3.11.9 and TensorFlow 2.18.0
  
  import tensorflow as tf
  
  f = tf.constant([2.0, 1.0, 3.0, 5.0, 4.0], dtype=tf.float32)
  h = tf.constant([1.0, -1.0], dtype=tf.float32)
  
  # Reshaping the inputs because conv1d accepts only certain shapes.
  f = tf.reshape(f, [1, int(f.shape[0]), 1])
  h = tf.reshape(h, [int(h.shape[0]), 1, 1])
  
  c = tf.nn.conv1d(f, h, stride=1, padding="VALID")
  
  # Using tf.squeeze to improve readability.
  print(f"conv1d of f = {tf.squeeze(f)} and h = {tf.squeeze(h)} = {tf.squeeze(c)}")
  
  ```
  
  Result
  
  ```
  conv1d of f = [2. 1. 3. 5. 4.] and h = [ 1. -1.] = [ 1. -2. -2.  1.]
  
  ```
  
  ## Further reading
  
  After having written this answer, I found the article [Convolution vs. Cross-Correlation](https://glassboxmedicine.com/2019/07/26/convolution-vs-cross-correlation) (2019) by Rachel Draelos, which essentially says the same thing that I am saying here, but provides more details and examples.
  
  
  ---
  
  ### Answer by username_2 on 2020-06-19. Score: 0
  
  Just as a short and quick answer to build of nbros:
  
  The way CNNs are typically taught, they are taught using a correlation on the forward pass, rather than a convolution. In reality, Convolutional neural networks is a bit of a misleading name, but not entirely incorrect.
  
  CNNs do in fact use convolutions every time they are trained and run. If a *correlation* is used on the forward pass, a *convolution* is used on the backward pass. The opposite is true if a convolution is used on the forward pass (which is equally valid as using a correlation). I couldn't seem to find this information anywhere, so had to learn it myself the hard way.
  
  So to summarise, a typical CNN goes like this: Correlation forward, convolution backward.
  
  > **Comments:**
  > **username_2**: I'd also like to say this a really good question and a pretty important one for anyone who wants to properly understand the inner workings of a CNN. It can be very confusing when you go to do an actual convolution and find it's not doing what you expect.  
  > **username_0**: Maybe you should explain why if the forward pass performs a convolution (or cross-correlation) the backward pass performs a cross-correlation (and, respectively, convolution). It was also my intention (although I haven't yet done it) to provide links to libraries and explain how they implement CNNs (in particular, the convolutional layers). If you know how certain libraries implement the CNNs (e.g. TensorFlow, PyTorch, etc.), then feel free to add this info to your answer.  
  > **username_0**: Also, I don't think that convolutional neural network is a misleading name. You could have called them "cross-correlation neural networks" (but that would be long and without any benefit) or, to be more precise, "convolutional or cross-correlation, down- or/and up-sampling, fully or not, with possible other layers, such as skip layers, neural networks", but that name would be quite ridiculous.  
  > **username_2**: @username_0 I still think it's slightly misleading in the sense that it's very easy to assume that a convolution is being performed when calculating an output, when in reality it's normally a correlation. Given that this is the core element that makes a CNN what it is, it can be a bit confusing. I know I personally found it confusing.  
  > **username_2**: @username_0 As for why for the relationship between correlation/convolution and how libraries implement CNNs, this I don't know. I figured out the correlation forward convolution backward by just mucking around and doing a few forward and backward passes by hand, and I haven't much looked into the implementation within libraries such as Tensorflow or Pytorch.  
  
  ---
  
  