Answers.Body: <p>Neural network is a generic term used in literature as a sort of umbrella for all types of architectures, architecture being a set of distinct forward operations and hyper parameters (such as number of layers/nodes, kernels size).</p>
  <p>Feedforward neural networks, multi layer perceptrons, convolutional neural networks, recurrent neural networks, autoencoders, transformers (and many more) are all types of neural networks (deep neural networks to be precise, the 'deep' is usually assumed). Also edge cases like generative adversarial networks (which is more of a training approach than a strict architecture) are usually referred to as neural network, which allegedly might be confusing.</p>
  <p>So &quot;<em>since transformers contain a neural network</em>&quot; is not really a correct way of putting it. Also in case you meant &quot;<em>since transformers contain a feed forward neural network</em>&quot; it would still be incorrect cause transformers use operations such as convolutions which are not used in feed forward neural networks, so they are still very distinct type of architectures.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-21T09:32:48.963Z
Answers.LastActivityDate: 2022-09-21T09:32:48.963Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>Transformers can be used for a variety of tasks that standard feedforward neural networks <span class="math-container">$\color{gray}{\textbf{cannot}}$</span>, such as <em>Natural Language Processing</em> and <em>Time Series Forecasting</em> &amp; they are <span class="math-container">$\color{gray}{\textbf{more efficient}}$</span> than standard feedforward neural networks, as they can share parameters across multiple tasks.</p>
  <p><span class="math-container">$\color{maroon}{\textbf{No}}$</span>, Transformers are not a <strong>Strict</strong> generalization of standard feedforward neural networks. Since there is <code>no</code> <strong>Strict</strong> definition of <code>what a &quot;standard feedforward neural network&quot; is</code>, but they <em>Can be seen as a Generalization of a Feed-Forward Neural Network in Several ways</em>.</p>
  <blockquote>
  <p>Transformers can be interpreted as a generalization of <em>recurrent neural networks (RNNs)</em>. In contrast to RNNs, which propagate hidden states through time, Transformers propagate hidden states through a self-attention mechanism. This allows them to model long-range dependencies without the need for RNNs, which are difficult to train.</p>
  </blockquote>
  <blockquote>
  <p>Transformers can also be interpreted as a generalization of <em>convolutional neural networks (CNNs)</em>. In contrast to CNNs, which extract features from local patches of an input, Transformers extract features from the entire input sequence. This allows them to model global dependencies without the need for CNNs, which are limited by their local receptive fields.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-21T13:46:02.037Z
Answers.LastActivityDate: 2022-10-21T13:46:02.037Z
Answers.UserDisplayName: Faizy
Body: <p>Since transformers contain a neural network, are they a strict generalisation of standard feedforward neural networks? In what ways can transformers be interpreted as a generalisation and abstraction of these?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-09-21T12:02:44.490Z
Comments.Text: https://deepai.org/machine-learning-glossary-and-terms/transformer-neural-network
Comments.UserDisplayName: Tony Stewart EE75
Tags: <neural-networks><transformer><feedforward-neural-networks>
Title: Are neural networks a strict special case of a transformer?
UserDisplayName: Featherball
ThreadText: # Are neural networks a strict special case of a transformer?
  
  (Asked by: username_0 on 2022-09-21)
  
  Since transformers contain a neural network, are they a strict generalisation of standard feedforward neural networks? In what ways can transformers be interpreted as a generalisation and abstraction of these?
  
  **Tags:** neural-networks, transformer, feedforward-neural-networks
  
  > **Comments:**
  > **username_1**: https://deepai.org/machine-learning-glossary-and-terms/transformer-neural-network  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-09-21. Score: 2
  
  Neural network is a generic term used in literature as a sort of umbrella for all types of architectures, architecture being a set of distinct forward operations and hyper parameters (such as number of layers/nodes, kernels size).
  
  Feedforward neural networks, multi layer perceptrons, convolutional neural networks, recurrent neural networks, autoencoders, transformers (and many more) are all types of neural networks (deep neural networks to be precise, the 'deep' is usually assumed). Also edge cases like generative adversarial networks (which is more of a training approach than a strict architecture) are usually referred to as neural network, which allegedly might be confusing.
  
  So "*since transformers contain a neural network*" is not really a correct way of putting it. Also in case you meant "*since transformers contain a feed forward neural network*" it would still be incorrect cause transformers use operations such as convolutions which are not used in feed forward neural networks, so they are still very distinct type of architectures.
  
  
  ---
  
  ### Answer by username_3 on 2022-10-21. Score: 0
  
  Transformers can be used for a variety of tasks that standard feedforward neural networks $\color{gray}{\textbf{cannot}}$, such as *Natural Language Processing* and *Time Series Forecasting* & they are $\color{gray}{\textbf{more efficient}}$ than standard feedforward neural networks, as they can share parameters across multiple tasks.
  
  $\color{maroon}{\textbf{No}}$, Transformers are not a **Strict** generalization of standard feedforward neural networks. Since there is `no` **Strict** definition of `what a "standard feedforward neural network" is`, but they *Can be seen as a Generalization of a Feed-Forward Neural Network in Several ways*.
  
  > Transformers can be interpreted as a generalization of *recurrent neural networks (RNNs)*. In contrast to RNNs, which propagate hidden states through time, Transformers propagate hidden states through a self-attention mechanism. This allows them to model long-range dependencies without the need for RNNs, which are difficult to train.
  
  > Transformers can also be interpreted as a generalization of *convolutional neural networks (CNNs)*. In contrast to CNNs, which extract features from local patches of an input, Transformers extract features from the entire input sequence. This allows them to model global dependencies without the need for CNNs, which are limited by their local receptive fields.
  
  
  ---
  
  