Answers.Body: <h3>What is a transformer?</h3>
  <p>The original <strong>transformer</strong>, proposed in the paper <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention is all you need</a> (2017), is an <strong>encoder-decoder</strong>-based neural network that is mainly characterized by the use of the so-called <a href="https://arxiv.org/pdf/1902.02181.pdf" rel="nofollow noreferrer"><strong>attention</strong></a> (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the <strong>non</strong>-use of recurrent connections (or recurrent neural networks) to solve tasks that involve sequences (or sentences), even though RNN-based systems were becoming the standard practice to solve natural language processing (NLP) or understanding (NLU) tasks. Hence the name of the paper &quot;<strong>Attention</strong> is all you need&quot;, i.e. you only need attention and you don't need recurrent connections to solve NLP tasks.</p>
  <p>Both the encoder-decoder architecture and the attention mechanism are not novel proposals. In fact, previous neural network architectures to solve many NLP tasks, such as machine translation, had already used these mechanisms (for example, take a look at <a href="https://arxiv.org/pdf/1409.0473.pdf" rel="nofollow noreferrer">this paper</a>).  The novelty of the transformer and <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">this cited paper</a> is that it shows that we can simply use attention to solve tasks that involve sequences (such as machine translation) and we do not need recurrent connections, which is an advantage, given that recurrent connections can hinder the parallelization of the training process.</p>
  <p>The original transformer architecture is depicted in figure 1 of <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">the cited paper</a>. Both the encoder and decoder are composed of</p>
  <ul>
  <li>attention modules</li>
  <li>feed-forward (or fully connected) layers</li>
  <li>residual (or skip) connections</li>
  <li>normalization layers</li>
  <li>dropout</li>
  <li>label smoothing</li>
  <li>embedding layers</li>
  <li>positional encoding</li>
  </ul>
  <p>The decoder part is also composed of a linear layer followed by a softmax to solve the specific NLP task (for example, predict the next word in a sentence).</p>
  <h3>What is BERT?</h3>
  <p><a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer"><strong>BERT</strong></a> stands for <em>Bidirectional <strong>Encoder Representations</strong> from
  <strong>Transformers</strong></em>, so, as the name suggests, it is a way of learning <strong>representations</strong> of a language that uses a <strong>transformer</strong>, specifically, the <strong>encoder</strong> part of the transformer.</p>
  <h3>What is the difference between the transformer and BERT?</h3>
  <ul>
  <li><p>BERT is a language model, i.e. it represents the statistical relationships of the words in a language, i.e. which words are more likely to come after another word and stuff like that. Hence the part <strong>Representations</strong> in its name, <em>Bidirectional Encoder <strong>Representations</strong> from
  Transformers</em>.</p>
  <p>BERT can be trained in an unsupervised way for <em>representation learning</em>, and then we can <em>fine-tune</em> BERT on the so-called downstream tasks in a supervised fashion (i.e. transfer learning). There are pre-trained versions of BERT that can be already fine-tuned (e.g. <a href="https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2" rel="nofollow noreferrer">this one</a>) and used to solve your specific supervised learning task. You can play with <a href="https://www.tensorflow.org/official_models/fine_tuning_bert" rel="nofollow noreferrer">this TensorFlow tutorial</a> to use a pre-trained BERT model.</p>
  <p>On the other hand, the original transformer was not originally conceived to be a language model, but to solve sequence transduction tasks (i.e. converting one sequence to another, such as machine translation) without recurrent connections (or convolutions) but only attention.</p>
  </li>
  <li><p>BERT is only an encoder, while the <em>original</em> transformer is composed of an encoder and decoder. Given that BERT uses an encoder that is very similar to the original encoder of the transformer, we can say that BERT is a transformer-based model. So, BERT does not use recurrent connections, but only attention and feed-forward layers. There are other transformer-based neural networks that use only the decoder part of the transformer, for example, <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf" rel="nofollow noreferrer">the GPT model</a>.</p>
  </li>
  <li><p>BERT uses different hyper-parameters than the ones used in <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention is all you need</a> to achieve the best performance. For example, it uses 12 and 16 &quot;attention heads&quot; (please, read the <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">transformer paper</a> to know more about these &quot;attention heads&quot;) rather than 8 (although in the original transformer paper the authors experimented with a different number of heads).</p>
  </li>
  <li><p>BERT also uses <em>segment embeddings</em>, while the original transformer only uses word embeddings and positional encodings.</p>
  </li>
  </ul>
  <p>There are probably other small differences that I missed, but, after having read the paper <a href="https://arxiv.org/pdf/1706.03762.pdf" rel="nofollow noreferrer">Attention is all you need</a> and quickly read some parts of <a href="https://arxiv.org/pdf/1810.04805.pdf" rel="nofollow noreferrer">the BERT paper</a>, these seem to be the main differences.</p>
  <h3>When to use BERT and the transformer?</h3>
  <p>Although I never used them, I would say that you want to use BERT whenever you want to solve an NLP task in a supervised fashion, but your labeled training dataset is not big enough to achieve good performance. In that case, you start with a pre-trained BERT model, then fine-tune it with your small labeled dataset. You probably need to add specific layers to BERT to solve your task.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-25T22:00:23.740Z
Answers.Text: Well explained!
Answers.UserDisplayName: avocado
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-13T23:57:42.303Z
Answers.Text: You said it was supposed to solve seq2seq tasks without convolution, but it does use convolution doesn't it? Right after the attention mechanism dense layers are convolved across the attention outputs + positional embeddings, albeit not with the traditional conventions of CNNs (e.g. there is no flexible stride).
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-14T08:54:39.067Z
Answers.Text: @profPlum There are no convolutions in the [transformer](https://arxiv.org/pdf/1706.03762.pdf). The original paper even says "Since our model contains no recurrence and no convolution...".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-18T23:35:32.280Z
Answers.Text: @nbro There are convolutions, though the authors call them "Point-wise feed-forward networks" and they discard the traditional conventions of CNNs (hence your quote).
  
  BUT the original paper *also said this*:
  "Another way of describing this [point-wise feedforward network] is as two convolutions with kernel size 1."
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-19T00:01:32.290Z
Answers.Text: @profPlum Thanks for sharing this info. As you can imagine, I gave this answer already a long time ago, so I forgot the details of the paper. However, note that you could also view a "normal" fully connected layer as a convolution layer with a kernel that has the same dimensions as the input. See also [this](https://ai.stackexchange.com/a/21824/2444).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-21T16:31:35.410Z
Answers.Text: @nbro Perhaps, but IIRC the point-wise FFNs 'slide' across the time-dimension (https://paperswithcode.com/method/position-wise-feed-forward-layer), which is perhaps a meaningful distinction.
  
  But ya, I can't blame you. ...The paper itself could stand to be much clearer on the matter haha
Answers.UserDisplayName: profPlum
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-21T16:04:08.137Z
Answers.LastActivityDate: 2024-05-31T02:46:02.070Z
Answers.LastEditDate: 2024-05-31T02:46:02.070Z
Answers.UserDisplayName: nbro
Body: <p>As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is:</p>
  <ol>
  <li><p>How is Bert different from the original transformer architecture?</p>
  </li>
  <li><p>What tasks are better suited for BERT, and what tasks are better suited for the original architecture?</p>
  </li>
  </ol>
  
Tags: <natural-language-processing><comparison><transformer><bert>
Title: How is BERT different from the original transformer architecture?
UserDisplayName: chessprogrammer
ThreadText: # How is BERT different from the original transformer architecture?
  
  (Asked by: username_0 on 2020-08-24)
  
  As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is:
  
  1. How is Bert different from the original transformer architecture?
  2. What tasks are better suited for BERT, and what tasks are better suited for the original architecture?
  
  **Tags:** natural-language-processing, comparison, transformer, bert
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-21. Score: 32 | Accepted answer
  
  ### What is a transformer?
  
  The original **transformer**, proposed in the paper [Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf) (2017), is an **encoder-decoder**-based neural network that is mainly characterized by the use of the so-called [**attention**](https://arxiv.org/pdf/1902.02181.pdf) (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the **non**-use of recurrent connections (or recurrent neural networks) to solve tasks that involve sequences (or sentences), even though RNN-based systems were becoming the standard practice to solve natural language processing (NLP) or understanding (NLU) tasks. Hence the name of the paper "**Attention** is all you need", i.e. you only need attention and you don't need recurrent connections to solve NLP tasks.
  
  Both the encoder-decoder architecture and the attention mechanism are not novel proposals. In fact, previous neural network architectures to solve many NLP tasks, such as machine translation, had already used these mechanisms (for example, take a look at [this paper](https://arxiv.org/pdf/1409.0473.pdf)). The novelty of the transformer and [this cited paper](https://arxiv.org/pdf/1706.03762.pdf) is that it shows that we can simply use attention to solve tasks that involve sequences (such as machine translation) and we do not need recurrent connections, which is an advantage, given that recurrent connections can hinder the parallelization of the training process.
  
  The original transformer architecture is depicted in figure 1 of [the cited paper](https://arxiv.org/pdf/1706.03762.pdf). Both the encoder and decoder are composed of
  
  * attention modules
  * feed-forward (or fully connected) layers
  * residual (or skip) connections
  * normalization layers
  * dropout
  * label smoothing
  * embedding layers
  * positional encoding
  
  The decoder part is also composed of a linear layer followed by a softmax to solve the specific NLP task (for example, predict the next word in a sentence).
  
  ### What is BERT?
  
  [**BERT**](https://arxiv.org/pdf/1810.04805.pdf) stands for *Bidirectional **Encoder Representations** from
  **Transformers***, so, as the name suggests, it is a way of learning **representations** of a language that uses a **transformer**, specifically, the **encoder** part of the transformer.
  
  ### What is the difference between the transformer and BERT?
  
  * BERT is a language model, i.e. it represents the statistical relationships of the words in a language, i.e. which words are more likely to come after another word and stuff like that. Hence the part **Representations** in its name, *Bidirectional Encoder **Representations** from
    Transformers*.
  
    BERT can be trained in an unsupervised way for *representation learning*, and then we can *fine-tune* BERT on the so-called downstream tasks in a supervised fashion (i.e. transfer learning). There are pre-trained versions of BERT that can be already fine-tuned (e.g. [this one](https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/2)) and used to solve your specific supervised learning task. You can play with [this TensorFlow tutorial](https://www.tensorflow.org/official_models/fine_tuning_bert) to use a pre-trained BERT model.
  
    On the other hand, the original transformer was not originally conceived to be a language model, but to solve sequence transduction tasks (i.e. converting one sequence to another, such as machine translation) without recurrent connections (or convolutions) but only attention.
  * BERT is only an encoder, while the *original* transformer is composed of an encoder and decoder. Given that BERT uses an encoder that is very similar to the original encoder of the transformer, we can say that BERT is a transformer-based model. So, BERT does not use recurrent connections, but only attention and feed-forward layers. There are other transformer-based neural networks that use only the decoder part of the transformer, for example, [the GPT model](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf).
  * BERT uses different hyper-parameters than the ones used in [Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf) to achieve the best performance. For example, it uses 12 and 16 "attention heads" (please, read the [transformer paper](https://arxiv.org/pdf/1706.03762.pdf) to know more about these "attention heads") rather than 8 (although in the original transformer paper the authors experimented with a different number of heads).
  * BERT also uses *segment embeddings*, while the original transformer only uses word embeddings and positional encodings.
  
  There are probably other small differences that I missed, but, after having read the paper [Attention is all you need](https://arxiv.org/pdf/1706.03762.pdf) and quickly read some parts of [the BERT paper](https://arxiv.org/pdf/1810.04805.pdf), these seem to be the main differences.
  
  ### When to use BERT and the transformer?
  
  Although I never used them, I would say that you want to use BERT whenever you want to solve an NLP task in a supervised fashion, but your labeled training dataset is not big enough to achieve good performance. In that case, you start with a pre-trained BERT model, then fine-tune it with your small labeled dataset. You probably need to add specific layers to BERT to solve your task.
  
  > **Comments:**
  > **username_2**: Well explained!  
  > **username_3**: You said it was supposed to solve seq2seq tasks without convolution, but it does use convolution doesn't it? Right after the attention mechanism dense layers are convolved across the attention outputs + positional embeddings, albeit not with the traditional conventions of CNNs (e.g. there is no flexible stride).  
  > **username_1**: @profPlum There are no convolutions in the [transformer](https://arxiv.org/pdf/1706.03762.pdf). The original paper even says "Since our model contains no recurrence and no convolution...".  
  > **username_3**: @nbro There are convolutions, though the authors call them "Point-wise feed-forward networks" and they discard the traditional conventions of CNNs (hence your quote).  BUT the original paper *also said this*: "Another way of describing this [point-wise feedforward network] is as two convolutions with kernel size 1."  
  > **username_1**: @profPlum Thanks for sharing this info. As you can imagine, I gave this answer already a long time ago, so I forgot the details of the paper. However, note that you could also view a "normal" fully connected layer as a convolution layer with a kernel that has the same dimensions as the input. See also [this](https://ai.stackexchange.com/a/21824/2444).  
  > **username_3**: @nbro Perhaps, but IIRC the point-wise FFNs 'slide' across the time-dimension (https://paperswithcode.com/method/position-wise-feed-forward-layer), which is perhaps a meaningful distinction.  But ya, I can't blame you. ...The paper itself could stand to be much clearer on the matter haha  
  
  ---
  
  