Answers.Body: <h2>What is a neural network?</h2>
  <p>Many neural networks can be defined as a function <span class="math-container">$f: \mathbb{R}^n \rightarrow \mathbb{R}^m$</span>, where <span class="math-container">$n, m \geq 1$</span>.</p>
  <p>Equivalently, many neural networks can also be defined as a set of interconnected units (aka neurons or nodes) <span class="math-container">$f^i$</span> that receive some input and produce output, i.e. <span class="math-container">$f^i(\mathbf{x}^i) = y^i$</span>, where <span class="math-container">$\mathbf{x}^i \in \mathbb{R}^k$</span>. The actual function <span class="math-container">$f^i$</span> is variable and depends on the application or problem you want to solve. For example, <span class="math-container">$f^i$</span> can just be a linear combination of the inputs, i.e. <span class="math-container">$f^i(\mathbf{x}^i) = \sum_{j} \mathbf{w}_j^i \mathbf{x}_{j}^i$</span>, where <span class="math-container">$\mathbf{w}^i \in \mathbb{R}^k$</span> is a vector of weights (aka parameters or coefficients). The linear combination can also be followed by a non-linear function, such as a sigmoid function.</p>
  <p>If the neural network (more precisely, its units) doesn't contain recurrent (aka cyclic) connections, then it can be called a feedforward neural network (FFNN).</p>
  <h2>What are the most common feedforward neural networks?</h2>
  <h3>Perceptron</h3>
  <p>The simplest (non-binary) FFNN is the <strong>perceptron</strong>, where the inputs are directly connected to the outputs. The perceptron performs a linear combination of the inputs followed by a thresholding operation, so perceptrons can only represent straight-line functions, so they can only be used for classification or regression problems where your data is linearly separable. In fact, the perceptron cannot solve the XOR problem.</p>
  <p>Before the perceptron, <a href="http://www.mind.ilstu.edu/curriculum/modOverview.php?modGUI=212" rel="nofollow noreferrer">McCulloch and Pitts</a> had introduced simplified models of biological neurons, where all signals are binary, in an attempt to closely mimic their biological counterpart. The perceptron can actually be seen as an extension of this work. In fact, a perceptron can be viewed as a single artificial neuron.</p>
  <h3>Multi-layer perceptron</h3>
  <p>An FFNN with more layers (of units) between the input and the output is often called a <strong>multi-layer perceptron</strong> (MLP). The layers in the middle are often denoted as <em>hidden layers</em>. The MLP can represent not only linear functions (i.e. straight lines), but also more complicated functions by making use of non-linear functions, such as the sigmoid.</p>
  <h3>Convolution neural network</h3>
  <p>You can have other forms of FFNNs that perform other operations.</p>
  <p>For example, a <strong>convolution neural network</strong> (CNN), provided it doesn't contain recurrent connections, is an FFNN that performs the <a href="https://ai.stackexchange.com/a/22000/2444">convolution operation</a> (and often also a sub-sampling operation). For this reason, they are particularly suited to deal with images (and videos). (This shouldn't be surprising if you are familiar with the basics of image processing and computer vision, which I don't think it's the case)</p>
  <p>However, note that CNNs can also have recurrent connections, but this is not usually the case.</p>
  <h3>Residual neural network</h3>
  <p>There are also <strong>residual neural networks</strong>, i.e. neural networks where a node in a certain layer <span class="math-container">$l$</span> can be connected to other nodes in layers <span class="math-container">$l+j$</span>, for <span class="math-container">$j \geq 1$</span>, as opposed to being connected only to the nodes in layer <span class="math-container">$l+1$</span>, which is the typical case.</p>
  <h3>Auto-encoders</h3>
  <p>Auto-encoders are neural networks that compress the input and then decompress it. The answers to <a href="https://ai.stackexchange.com/q/11405/2444">this question</a> may help you to understand why AEs would be useful.</p>
  <h2>What kind of inputs do they receive?</h2>
  <blockquote>
  <p>What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics?</p>
  </blockquote>
  <p>In principle, each of these FFNNs can receive either binary or real numbers or vectors (either of real or binary numbers). However, certain NNs are more appropriate to deal with certain inputs. For example, CNNs are more appropriate for images (which are typically represented as matrices or tensors).</p>
  <h2>How can you further classify the NNs?</h2>
  <p>Based on <a href="https://page.mi.fu-berlin.de/rojas/neural/chapter/K2.pdf" rel="nofollow noreferrer">chapter 2 of the book Neural Networks - A Systematic Introduction</a> (1996) by Raul Rojas, you can also divide neural networks into other categories</p>
  <ul>
  <li>Unweighted (i.e. binary, such as the McCulloch and Pitts' model) vs weighted (e.g. the perceptron)</li>
  <li>Synchronous vs asynchronous (e.g. Hopfield networks, which are recurrent neural networks, though)</li>
  <li>Neural networks that store states vs NNs that don't store states</li>
  </ul>
  <p>You could also distinguish between FFNNs based on the learning algorithm. Nowadays, the widely used NNs are trained with gradient descent (and back-propagation to compute the gradients), but there are other approaches to train NNs, such as evolutionary algorithms or Hebbian learning. Moreover, you could also distinguish between neural networks that compute a deterministic function and neural networks that have some randomness or stochasticity inside them (e.g. Bayesian neural networks). There are probably many more possible subdivisions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-20T22:30:32.210Z
Answers.LastActivityDate: 2020-07-20T12:12:53.437Z
Answers.LastEditDate: 2020-07-20T12:12:53.437Z
Answers.UserDisplayName: nbro
Body: <p>What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy? </p>
  
Tags: <neural-networks><reference-request><feedforward-neural-networks>
Title: What are the most common feedforward neural networks?
UserDisplayName: Caesar Cruez
ThreadText: # What are the most common feedforward neural networks?
  
  (Asked by: username_0 on 2020-05-20)
  
  What are the most common feedforward neural networks? What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics? Is there such a taxonomy?
  
  **Tags:** neural-networks, reference-request, feedforward-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-20. Score: 1
  
  ## What is a neural network?
  
  Many neural networks can be defined as a function $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$, where $n, m \geq 1$.
  
  Equivalently, many neural networks can also be defined as a set of interconnected units (aka neurons or nodes) $f^i$ that receive some input and produce output, i.e. $f^i(\mathbf{x}^i) = y^i$, where $\mathbf{x}^i \in \mathbb{R}^k$. The actual function $f^i$ is variable and depends on the application or problem you want to solve. For example, $f^i$ can just be a linear combination of the inputs, i.e. $f^i(\mathbf{x}^i) = \sum\_{j} \mathbf{w}\_j^i \mathbf{x}\_{j}^i$, where $\mathbf{w}^i \in \mathbb{R}^k$ is a vector of weights (aka parameters or coefficients). The linear combination can also be followed by a non-linear function, such as a sigmoid function.
  
  If the neural network (more precisely, its units) doesn't contain recurrent (aka cyclic) connections, then it can be called a feedforward neural network (FFNN).
  
  ## What are the most common feedforward neural networks?
  
  ### Perceptron
  
  The simplest (non-binary) FFNN is the **perceptron**, where the inputs are directly connected to the outputs. The perceptron performs a linear combination of the inputs followed by a thresholding operation, so perceptrons can only represent straight-line functions, so they can only be used for classification or regression problems where your data is linearly separable. In fact, the perceptron cannot solve the XOR problem.
  
  Before the perceptron, [McCulloch and Pitts](http://www.mind.ilstu.edu/curriculum/modOverview.php?modGUI=212) had introduced simplified models of biological neurons, where all signals are binary, in an attempt to closely mimic their biological counterpart. The perceptron can actually be seen as an extension of this work. In fact, a perceptron can be viewed as a single artificial neuron.
  
  ### Multi-layer perceptron
  
  An FFNN with more layers (of units) between the input and the output is often called a **multi-layer perceptron** (MLP). The layers in the middle are often denoted as *hidden layers*. The MLP can represent not only linear functions (i.e. straight lines), but also more complicated functions by making use of non-linear functions, such as the sigmoid.
  
  ### Convolution neural network
  
  You can have other forms of FFNNs that perform other operations.
  
  For example, a **convolution neural network** (CNN), provided it doesn't contain recurrent connections, is an FFNN that performs the [convolution operation](https://ai.stackexchange.com/a/22000/2444) (and often also a sub-sampling operation). For this reason, they are particularly suited to deal with images (and videos). (This shouldn't be surprising if you are familiar with the basics of image processing and computer vision, which I don't think it's the case)
  
  However, note that CNNs can also have recurrent connections, but this is not usually the case.
  
  ### Residual neural network
  
  There are also **residual neural networks**, i.e. neural networks where a node in a certain layer $l$ can be connected to other nodes in layers $l+j$, for $j \geq 1$, as opposed to being connected only to the nodes in layer $l+1$, which is the typical case.
  
  ### Auto-encoders
  
  Auto-encoders are neural networks that compress the input and then decompress it. The answers to [this question](https://ai.stackexchange.com/q/11405/2444) may help you to understand why AEs would be useful.
  
  ## What kind of inputs do they receive?
  
  > What kind of inputs do they receive? For example, do they receive binary numbers, real numbers, vectors, or matrics?
  
  In principle, each of these FFNNs can receive either binary or real numbers or vectors (either of real or binary numbers). However, certain NNs are more appropriate to deal with certain inputs. For example, CNNs are more appropriate for images (which are typically represented as matrices or tensors).
  
  ## How can you further classify the NNs?
  
  Based on [chapter 2 of the book Neural Networks - A Systematic Introduction](https://page.mi.fu-berlin.de/rojas/neural/chapter/K2.pdf) (1996) by Raul Rojas, you can also divide neural networks into other categories
  
  * Unweighted (i.e. binary, such as the McCulloch and Pitts' model) vs weighted (e.g. the perceptron)
  * Synchronous vs asynchronous (e.g. Hopfield networks, which are recurrent neural networks, though)
  * Neural networks that store states vs NNs that don't store states
  
  You could also distinguish between FFNNs based on the learning algorithm. Nowadays, the widely used NNs are trained with gradient descent (and back-propagation to compute the gradients), but there are other approaches to train NNs, such as evolutionary algorithms or Hebbian learning. Moreover, you could also distinguish between neural networks that compute a deterministic function and neural networks that have some randomness or stochasticity inside them (e.g. Bayesian neural networks). There are probably many more possible subdivisions.
  
  
  ---
  
  