Answers.Body: <p><strong>TLDR:</strong>
  The convolutional-neural-network is a subclass of neural-networks which have at least one convolution layer. They are great for capturing local information (e.g. neighbor pixels in an image or surrounding words in a text) as well as reducing the complexity of the model (faster training, needs fewer samples, reduces the chance of overfitting).</p>
  <p>See the following chart that depicts the several neural-networks architectures including deep-conventional-neural-networks: <a href="https://i.sstatic.net/LgmYv.png" rel="noreferrer"><img src="https://i.sstatic.net/LgmYv.png" alt="visualization" /></a>.</p>
  <hr />
  <p><strong>Neural Networks (NN)</strong>, or more precisely <strong>Artificial Neural Networks (ANN)</strong>, is a class of Machine Learning algorithms that recently received a lot of attention (again!) due to the availability of Big Data and fast computing facilities (most of Deep Learning algorithms are essentially different variations of ANN).</p>
  <p>The class of ANN covers several architectures including <strong>Convolutional Neural Networks</strong> (<a href="https://en.wikipedia.org/wiki/Convolutional_neural_network" rel="noreferrer">CNN</a>), Recurrent Neural Networks (<a href="https://en.wikipedia.org/wiki/Recurrent_neural_network" rel="noreferrer">RNN</a>) eg <a href="https://en.wikipedia.org/wiki/Long_short-term_memory" rel="noreferrer">LSTM</a> and <a href="https://en.wikipedia.org/wiki/Gated_recurrent_unit" rel="noreferrer">GRU</a>, <a href="https://en.wikipedia.org/wiki/Autoencoder" rel="noreferrer">Autoencoders</a>, and <a href="https://en.wikipedia.org/wiki/Deep_belief_network" rel="noreferrer">Deep Belief Networks</a>. Therefore, CNN is just one kind of ANN.</p>
  <p>Generally speaking, an ANN is a collection of connected and tunable units (a.k.a. nodes, neurons, and artificial neurons) which can pass a signal (usually a real-valued number) from a unit to another. The number of (layers of) units, their types, and the way they are connected to each other is called the network architecture.</p>
  <p>A CNN, in specific, has one or more layers of <em><strong>convolution</strong></em> units. A convolution unit receives its input from multiple units from the previous layer which together create a proximity. Therefore, the input units (that form a small neighborhood) share their weights.</p>
  <p>The convolution units (as well as pooling units) are especially beneficial as:</p>
  <ul>
  <li>They reduce the number of units in the network (since they are <em>many-to-one mappings</em>). This means, there are fewer parameters to learn which reduces the chance of overfitting as the model would be less complex than a fully connected network.</li>
  <li>They consider the context/shared information in the small neighborhoods. This feature is very important in many applications such as image, video, text, and speech processing/mining as the neighboring inputs (eg pixels, frames, words, etc) usually carry related information.</li>
  </ul>
  <p>Read the followings for more information about (deep) CNNs:</p>
  <ol>
  <li><a href="https://www.cs.toronto.edu/%7Ekriz/imagenet_classification_with_deep_convolutional.pdf" rel="noreferrer">ImageNet Classification with Deep Convolutional Neural Networks</a></li>
  <li><a href="https://arxiv.org/abs/1409.4842" rel="noreferrer">Going Deeper with Convolutions</a></li>
  </ol>
  <hr />
  <p>P.S. ANN is not <em>&quot;a system based loosely on the human brain&quot;</em> but rather a <strong>class</strong> of systems <strong>inspired</strong> by the neuron connections exist in animal brains.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-08T01:49:51.950Z
Answers.LastActivityDate: 2022-06-27T16:14:08.993Z
Answers.LastEditDate: 2022-06-27T16:14:08.993Z
Answers.UserDisplayName: Borhan Kazimipour
Answers.Body: <p>Convolutional Neural Networks (CNNs) are neural networks with architectural constraints to reduce computational complexity and ensure translational invariance (the network interprets input patterns the same regardless of translation— in terms of image recognition: a banana is a banana regardless of where it is in the image). Convolutional Neural Networks have three important architectural features.</p>
  
  <p><strong><em>Local Connectivity:</em></strong> Neurons in one layer are only connected to neurons in the next layer that are spatially close to them. This design trims the vast majority of connections between consecutive layers, but keeps the ones that carry the most useful information. The assumption made here is that the input data has spatial significance, or in the example of computer vision, the relationship between two distant pixels is probably less significant than two close neighbors.</p>
  
  <p><strong><em>Shared Weights:</em></strong> This is the concept that makes CNNs "convolutional." By forcing the neurons of one layer to share weights, the forward pass (feeding data through the network) becomes the equivalent of convolving a filter over the image to produce a new image. The training of CNNs then becomes the task of learning filters (deciding what features you should look for in the data.)</p>
  
  <p><strong><em>Pooling and ReLU:</em></strong> CNNs have two non-linearities: pooling layers and ReLU functions. Pooling layers consider a block of input data and simply pass on the maximum value. Doing this reduces the size of the output and requires no added parameters to learn, so pooling layers are often used to regulate the size of the network and keep the system below a computational limit. The ReLU function takes one input, x, and returns the maximum of {0, x}. <code>ReLU(x) = argmax(x, 0)</code>. This introduces a similar effect to tanh(x) or sigmoid(x) as non-linearities to increase the model's expressive power.</p>
  
  <hr>
  
  <h2>Further Reading</h2>
  
  <p>As another answer mentioned, Stanford's CS 231n course covers this in detail. Check out <a href="http://cs231n.github.io/convolutional-networks/#overview" rel="noreferrer">this written guide</a> and <a href="https://www.youtube.com/watch?v=bNb2fEVKeEo&amp;t=0s&amp;list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&amp;index=5" rel="noreferrer">this lecture</a> for more information. Blog posts like <a href="https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/" rel="noreferrer">this one</a> and <a href="https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/" rel="noreferrer">this one</a> are also very helpful.</p>
  
  <p>If you're still curious why CNNs have the structure that they do, I suggest reading <a href="http://www.dengfanxin.cn/wp-content/uploads/2016/03/1998Lecun.pdf" rel="noreferrer">the paper that introduced them</a> though this is quite long, and perhaps checking out <a href="https://www.youtube.com/watch?v=fKk9KhGRBdI" rel="noreferrer">this discussion</a> between Yann Lecun and Christopher Manning about innate priors (the assumptions we make when we design the architecture of a model).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T15:50:04.093Z
Answers.Text: This is a better answer to me in that it explains exactly how CNNs are a specific type of NN. Other answers don't mention that weight sharing is enforced.
Answers.UserDisplayName: Denziloe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-07T18:50:07.183Z
Answers.LastActivityDate: 2018-11-03T18:17:02.057Z
Answers.LastEditDate: 2018-11-03T18:17:02.057Z
Answers.UserDisplayName: Jackson Waschura
Answers.Body: <p>A <em>convolutional</em> neural network is one that has <em>convolutional</em> layers. If a general neural network is, loosely speaking, inspired by a human brain (which isn't very much accurate), the convolutional neural network is inspired by the visual cortex system, in humans and other animals (which is closer to the truth). As the name suggests, this layer applies the convolution with a learnable filter (a.k.a. <em>kernel</em>), as a result the network learns the patterns in the images: edges, corners, arcs, then more complex figures. Convolutional neural network may contain other layers as well, commonly pooling and dense layers.</p>
  
  <p>Highly recommend <a href="http://cs231n.github.io/convolutional-networks/#overview" rel="noreferrer">CS231n tutorial</a> on this matter: it's very detailed and contains a lot of very nice visualizations.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-06T23:01:38.160Z
Answers.LastActivityDate: 2018-03-06T23:01:38.160Z
Answers.UserDisplayName: Maxim
Answers.Body: <p>The everyday definition of <em>convolution</em> comes from the Latin <a href="https://en.wiktionary.org/wiki/convolution#English" rel="nofollow noreferrer"><em>convolutus</em></a> meaning 'to roll together'. Hence the meaning twisted or complicated.</p>
  
  <p>The mathematical definition comes from the same root, with the interpretation of taking a "rolling average".</p>
  
  <p>Hence in Machine Learning, a <em>convolution</em> is a sliding window across an input creating one averaged output for each stride the window takes. I.e. the values covered by the window are <em>convoluted</em> to create one convoluted output. This is best demonstrated with an a diagram:</p>
  
  <p><a href="https://i.sstatic.net/AV512.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AV512.gif" alt="enter image description here"></a></p>
  
  <p>The convolution can be any function of the input, but some common ones are the max value, or the mean value.</p>
  
  <p>A convolutional neural network (CNN) is a neural network where one or more of the layers employs a convolution as the function applied to the output of the previous layer.</p>
  
  <p>If the window is greater than size 1x1, the output will be necessarily smaller than the input (unless the input is artificially 'padded' with zeros), and hence CNN's often have a distinctive 'funnel' shape:</p>
  
  <p><a href="https://i.sstatic.net/p2gz1m.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/p2gz1m.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-22T13:37:02.537Z
Answers.Text: Although this is a good answer, I think, it does not mention anything about "regular neural networks". It seems that your answer would be more an answer to the question [here](https://ai.stackexchange.com/q/21957/2444). So, you may consider deleting it from here and pasting it there or edit it to include something about "regular NNs"
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-25T22:13:50.023Z
Answers.LastActivityDate: 2020-04-25T22:13:50.023Z
Answers.UserDisplayName: brazofuerte
Answers.Body: <p>It's a very simplified explantion. I am just talking about the core idea. </p>
  
  <p>A neural network is a combination of many layers.  </p>
  
  <p>A neural network (Multiple Layer Perceptron: Regular neural network ): It does a linear combination (a mathematical operation) between the previous layer's output and the current layer's weights(vectors) and then it passes data to the next layer by passing through an activation function. The picture shows a unit of a layer.</p>
  
  <p><a href="https://i.sstatic.net/YwCCU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YwCCU.png" alt="A normal neural network. w represent weight "></a></p>
  
  <p>A neural network (Convolutional Neural Network):  It does convolution (In signal processing it's known as Correlation) (Its a mathematical operation) between the previous layer's output and the current layer's kernel ( a small matrix ) and then it passes data to the next layer by passing through an activation function. The picture shows a Convolution operation. Each layer may have many convolution operation </p>
  
  <p><a href="https://i.sstatic.net/13ayz.gif" rel="nofollow noreferrer"><img src="https://i.sstatic.net/13ayz.gif" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-13T18:08:45.177Z
Answers.LastActivityDate: 2020-03-13T18:08:45.177Z
Answers.UserDisplayName: Ta_Req
Body: <p>I've seen these terms thrown around this site a lot, specifically in the tags <a href="/questions/tagged/convolutional-neural-networks" class="post-tag" title="show questions tagged &#39;convolutional-neural-networks&#39;" rel="tag">convolutional-neural-networks</a> and <a href="/questions/tagged/neural-networks" class="post-tag" title="show questions tagged &#39;neural-networks&#39;" rel="tag">neural-networks</a>.</p>
  
  <p>I know that a neural network is a system based loosely on the human brain. But what's the difference between a <em>convolutional</em> neural network and a regular neural network? Is one just a lot more complicated and, ahem, <em>convoluted</em> than the other? </p>
  
Tags: <neural-networks><convolutional-neural-networks><comparison><terminology><definitions>
Title: What is the difference between a convolutional neural network and a regular neural network?
UserDisplayName: Mithical
ThreadText: # What is the difference between a convolutional neural network and a regular neural network?
  
  (Asked by: username_0 on 2018-03-06)
  
  I've seen these terms thrown around this site a lot, specifically in the tags [convolutional-neural-networks](/questions/tagged/convolutional-neural-networks "show questions tagged 'convolutional-neural-networks'") and [neural-networks](/questions/tagged/neural-networks "show questions tagged 'neural-networks'").
  
  I know that a neural network is a system based loosely on the human brain. But what's the difference between a *convolutional* neural network and a regular neural network? Is one just a lot more complicated and, ahem, *convoluted* than the other?
  
  **Tags:** neural-networks, convolutional-neural-networks, comparison, terminology, definitions
  
  
  ---
  
  ## 5 answers
  
  ### Answer by username_1 on 2018-03-08. Score: 43
  
  **TLDR:**
  The convolutional-neural-network is a subclass of neural-networks which have at least one convolution layer. They are great for capturing local information (e.g. neighbor pixels in an image or surrounding words in a text) as well as reducing the complexity of the model (faster training, needs fewer samples, reduces the chance of overfitting).
  
  See the following chart that depicts the several neural-networks architectures including deep-conventional-neural-networks: [![visualization](https://i.sstatic.net/LgmYv.png)](https://i.sstatic.net/LgmYv.png).
  
  ---
  
  **Neural Networks (NN)**, or more precisely **Artificial Neural Networks (ANN)**, is a class of Machine Learning algorithms that recently received a lot of attention (again!) due to the availability of Big Data and fast computing facilities (most of Deep Learning algorithms are essentially different variations of ANN).
  
  The class of ANN covers several architectures including **Convolutional Neural Networks** ([CNN](https://en.wikipedia.org/wiki/Convolutional_neural_network)), Recurrent Neural Networks ([RNN](https://en.wikipedia.org/wiki/Recurrent_neural_network)) eg [LSTM](https://en.wikipedia.org/wiki/Long_short-term_memory) and [GRU](https://en.wikipedia.org/wiki/Gated_recurrent_unit), [Autoencoders](https://en.wikipedia.org/wiki/Autoencoder), and [Deep Belief Networks](https://en.wikipedia.org/wiki/Deep_belief_network). Therefore, CNN is just one kind of ANN.
  
  Generally speaking, an ANN is a collection of connected and tunable units (a.k.a. nodes, neurons, and artificial neurons) which can pass a signal (usually a real-valued number) from a unit to another. The number of (layers of) units, their types, and the way they are connected to each other is called the network architecture.
  
  A CNN, in specific, has one or more layers of ***convolution*** units. A convolution unit receives its input from multiple units from the previous layer which together create a proximity. Therefore, the input units (that form a small neighborhood) share their weights.
  
  The convolution units (as well as pooling units) are especially beneficial as:
  
  * They reduce the number of units in the network (since they are *many-to-one mappings*). This means, there are fewer parameters to learn which reduces the chance of overfitting as the model would be less complex than a fully connected network.
  * They consider the context/shared information in the small neighborhoods. This feature is very important in many applications such as image, video, text, and speech processing/mining as the neighboring inputs (eg pixels, frames, words, etc) usually carry related information.
  
  Read the followings for more information about (deep) CNNs:
  
  1. [ImageNet Classification with Deep Convolutional Neural Networks](https://www.cs.toronto.edu/%7Ekriz/imagenet_classification_with_deep_convolutional.pdf)
  2. [Going Deeper with Convolutions](https://arxiv.org/abs/1409.4842)
  
  ---
  
  P.S. ANN is not *"a system based loosely on the human brain"* but rather a **class** of systems **inspired** by the neuron connections exist in animal brains.
  
  
  ---
  
  ### Answer by username_2 on 2018-03-07. Score: 19
  
  Convolutional Neural Networks (CNNs) are neural networks with architectural constraints to reduce computational complexity and ensure translational invariance (the network interprets input patterns the same regardless of translation— in terms of image recognition: a banana is a banana regardless of where it is in the image). Convolutional Neural Networks have three important architectural features.
  
  ***Local Connectivity:*** Neurons in one layer are only connected to neurons in the next layer that are spatially close to them. This design trims the vast majority of connections between consecutive layers, but keeps the ones that carry the most useful information. The assumption made here is that the input data has spatial significance, or in the example of computer vision, the relationship between two distant pixels is probably less significant than two close neighbors.
  
  ***Shared Weights:*** This is the concept that makes CNNs "convolutional." By forcing the neurons of one layer to share weights, the forward pass (feeding data through the network) becomes the equivalent of convolving a filter over the image to produce a new image. The training of CNNs then becomes the task of learning filters (deciding what features you should look for in the data.)
  
  ***Pooling and ReLU:*** CNNs have two non-linearities: pooling layers and ReLU functions. Pooling layers consider a block of input data and simply pass on the maximum value. Doing this reduces the size of the output and requires no added parameters to learn, so pooling layers are often used to regulate the size of the network and keep the system below a computational limit. The ReLU function takes one input, x, and returns the maximum of {0, x}. `ReLU(x) = argmax(x, 0)`. This introduces a similar effect to tanh(x) or sigmoid(x) as non-linearities to increase the model's expressive power.
  
  ---
  
  ## Further Reading
  
  As another answer mentioned, Stanford's CS 231n course covers this in detail. Check out [this written guide](http://cs231n.github.io/convolutional-networks/#overview) and [this lecture](https://www.youtube.com/watch?v=bNb2fEVKeEo&t=0s&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=5) for more information. Blog posts like [this one](https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/) and [this one](https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/) are also very helpful.
  
  If you're still curious why CNNs have the structure that they do, I suggest reading [the paper that introduced them](http://www.dengfanxin.cn/wp-content/uploads/2016/03/1998Lecun.pdf) though this is quite long, and perhaps checking out [this discussion](https://www.youtube.com/watch?v=fKk9KhGRBdI) between Yann Lecun and Christopher Manning about innate priors (the assumptions we make when we design the architecture of a model).
  
  > **Comments:**
  > **username_3**: This is a better answer to me in that it explains exactly how CNNs are a specific type of NN. Other answers don't mention that weight sharing is enforced.  
  
  ---
  
  ### Answer by username_4 on 2018-03-06. Score: 6
  
  A *convolutional* neural network is one that has *convolutional* layers. If a general neural network is, loosely speaking, inspired by a human brain (which isn't very much accurate), the convolutional neural network is inspired by the visual cortex system, in humans and other animals (which is closer to the truth). As the name suggests, this layer applies the convolution with a learnable filter (a.k.a. *kernel*), as a result the network learns the patterns in the images: edges, corners, arcs, then more complex figures. Convolutional neural network may contain other layers as well, commonly pooling and dense layers.
  
  Highly recommend [CS231n tutorial](http://cs231n.github.io/convolutional-networks/#overview) on this matter: it's very detailed and contains a lot of very nice visualizations.
  
  
  ---
  
  ### Answer by username_5 on 2020-04-25. Score: 2
  
  The everyday definition of *convolution* comes from the Latin [*convolutus*](https://en.wiktionary.org/wiki/convolution#English) meaning 'to roll together'. Hence the meaning twisted or complicated.
  
  The mathematical definition comes from the same root, with the interpretation of taking a "rolling average".
  
  Hence in Machine Learning, a *convolution* is a sliding window across an input creating one averaged output for each stride the window takes. I.e. the values covered by the window are *convoluted* to create one convoluted output. This is best demonstrated with an a diagram:
  
  [![enter image description here](https://i.sstatic.net/AV512.gif)](https://i.sstatic.net/AV512.gif)
  
  The convolution can be any function of the input, but some common ones are the max value, or the mean value.
  
  A convolutional neural network (CNN) is a neural network where one or more of the layers employs a convolution as the function applied to the output of the previous layer.
  
  If the window is greater than size 1x1, the output will be necessarily smaller than the input (unless the input is artificially 'padded' with zeros), and hence CNN's often have a distinctive 'funnel' shape:
  
  [![enter image description here](https://i.sstatic.net/p2gz1m.png)](https://i.sstatic.net/p2gz1m.png)
  
  > **Comments:**
  > **username_6**: Although this is a good answer, I think, it does not mention anything about "regular neural networks". It seems that your answer would be more an answer to the question [here](https://ai.stackexchange.com/q/21957/2444). So, you may consider deleting it from here and pasting it there or edit it to include something about "regular NNs"  
  
  ---
  
  ### Answer by username_7 on 2020-03-13. Score: 1
  
  It's a very simplified explantion. I am just talking about the core idea.
  
  A neural network is a combination of many layers.
  
  A neural network (Multiple Layer Perceptron: Regular neural network ): It does a linear combination (a mathematical operation) between the previous layer's output and the current layer's weights(vectors) and then it passes data to the next layer by passing through an activation function. The picture shows a unit of a layer.
  
  [![A normal neural network. w represent weight ](https://i.sstatic.net/YwCCU.png)](https://i.sstatic.net/YwCCU.png)
  
  A neural network (Convolutional Neural Network): It does convolution (In signal processing it's known as Correlation) (Its a mathematical operation) between the previous layer's output and the current layer's kernel ( a small matrix ) and then it passes data to the next layer by passing through an activation function. The picture shows a Convolution operation. Each layer may have many convolution operation
  
  [![enter image description here](https://i.sstatic.net/13ayz.gif)](https://i.sstatic.net/13ayz.gif)
  
  
  ---
  
  