Answers.Body: <p>The <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">original transformer paper</a> presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a &quot;transformer model&quot;, but it actually consists only of an encoder or only of a decoder. That's fine as long as you know what exactly is going on.
  (I would say that the distinctive feature of the transformer model is the <a href="https://pi-tau.github.io/posts/transformer/#attention" rel="noreferrer">attention layer</a>.)</p>
  <p>For example, as you mentioned, <strong>G</strong>enerative <strong>P</strong>retrained <strong>Transformer</strong> (GPT) is a decoder-only model, and <strong>Vi</strong>sion <strong>Transformer</strong> (ViT) is an encoder-only model.</p>
  <p>I feel like your question already contains the answer :)
  Yes, encoders are used exactly for that - encoding (or compressing) the input. Note that the output sequence from the encoder has the same length as the input sequence. This is why encoder-only models cannot be used for tasks where the output sequence has a different length or unknown length, e.g. machine translation, text summarization. In these cases you need the decoder.</p>
  <p>Encoder-only models like ViT and BERT will encode the input sequence and produce an output sequence of the same length. Then for classification they will use only the first element of the output sequence.</p>
  <p><a href="https://i.sstatic.net/N0vDZ.png" rel="noreferrer"><img src="https://i.sstatic.net/N0vDZ.png" alt="Vision Transformer" /></a></p>
  <p>As I said, encoder-only models cannot be used for generating an output sequence of different (or unknown) length, i.e. BERT cannot be used for machine translation or generative question answering. BERT can only be used for question answering on <a href="https://rajpurkar.github.io/SQuAD-explorer/" rel="noreferrer">SQuAD</a>-type datasets, i.e. your answer is a continuous segment from the input and the model only outputs <code>&lt;START_IDX&gt;</code> and <code>&lt;END_IDX&gt;</code> to mark it.</p>
  <p>For generation tasks you need a decoder. The decoder is actually an auto-regressive model. It will generate the elements of the output sequence one-by-one until it decides that the sequence is ready and then it generates the final <code>&lt;END&gt;</code> token. See <a href="https://pi-tau.github.io/posts/transformer/#inference" rel="noreferrer">here</a>.</p>
  <p>Previously people used an encoder-decoder architecture to solve these sequence-to-sequence tasks (e.g. <a href="https://arxiv.org/pdf/1910.10683.pdf" rel="noreferrer">T5</a>). You want the encoder to encode your input sequence and the decoder to decode it and produce the output sequence. However, it turns out that you can use a decoder-only model. You simply concatenate your source and target sentences and treat the task as a language modelling task. Now there are pros and cons to this approach, but the main takeaway is that you can do it if you want to.</p>
  <p>You can checkout these papers to read more about encoder-decoder vs decoder-only models for sequence to sequence tasks:<br />
  <a href="https://arxiv.org/pdf/2304.04052.pdf" rel="noreferrer">https://arxiv.org/pdf/2304.04052.pdf</a><br />
  <a href="https://aclanthology.org/2022.aacl-main.43.pdf" rel="noreferrer">https://aclanthology.org/2022.aacl-main.43.pdf</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T15:55:55.713Z
Answers.Text: Thanks for helping about my question. I need more study with your answer. I still confused, because why **google bard**(based on bert, made with only transformer encoder) can do text summmarization on google services. I test google bard with my question, request like "please summarize about given text." and paste my question. The bard gives answer like.... (next comment)
Answers.UserDisplayName: Yang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T15:58:04.703Z
Answers.Text: Sure, I can help you with that. Here is the summary of the text you provided:
  
  -Transformer models consist of two main components: an encoder and a decoder.
  -The encoder takes in a sequence of input data and generates a fixed-length representation of it.
  -The decoder takes in this representation and generates an output sequence.
  -In some cases, only the encoder or only the decoder is used.
  -The encoder is often used for tasks such as classification, while the decoder is often used for tasks such as generation.
  -However, there are also cases where both the encoder and decoder are used.
Answers.UserDisplayName: Yang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-31T16:06:06.027Z
Answers.Text: And then **bard** give answer about my original question. As we know(according to Pi-tau give knowledge), only encoder model can't do summarization. But google Bard tried to summarize the text. why **bard** looks like **bard** can do summarization? I need more time to search more info, because I want to reask with more background. Thank for helping, and If others still don't write answer on this question, I accept your answer after 1 week. :)
Answers.UserDisplayName: Yang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T23:14:22.890Z
Answers.Text: Okay, so first of all Bard is not based on BERT -- It uses LaMDA (https://arxiv.org/pdf/2201.08239.pdf). You can see in the third paragraph of Sec. 3 from the paper that this is a decoder-only model, just like GPT. The difference is that it is pre-trained specifically on dialog data.  
  
  >"We use a decoder-only Transformer [ 92 ] language model as the model architecture for LaMDA."
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-01T23:21:28.757Z
Answers.Text: Please note also that Bard is not just a transformer decoder language model. It is a complex system consisting of **many**, **many** components, one of which is the transformer decoder. It is also connected to google's search engine, so it can search the web for information that can help it generate the correct answer. I would recommend this podcast if you want to hear more about how models are trained to use the web: https://www.talkrl.com/episodes/john-schulman.
Answers.UserDisplayName: pi-tau
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-09T13:58:25.857Z
Answers.Text: Thank you! It really help to me, then I understand about my question. :)
Answers.UserDisplayName: Yang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-01T08:55:33.250Z
Answers.Text: So, the difference is about seqlen where encoder is fixed and decoder is dynamic. In that case, encoder also can be used for summarization and generation, just set the fixed seqlen as much as possible (though it's impractical). Where we can also use stop token for both input and target, the dataset is simply the sliding window, where padding token is mandatory here.
Answers.UserDisplayName: Muhammad Ikhwan Perwira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-29T15:21:59.733Z
Answers.LastActivityDate: 2023-07-29T15:30:26.093Z
Answers.LastEditDate: 2023-07-29T15:30:26.093Z
Answers.UserDisplayName: pi-tau
Body: <p>I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.</p>
  <p>The overall form of a transformer consists of an encoder and a decoder. Depending on the model, you may use only the encoder, only the decoder, or both. However, for what purpose do model designers use only encoders, only decoders, or both?</p>
  <p>I already knew that encoders in transformers are as known as taking in a sequence of input data and generates a fixed-length representation of it. This representation can then be fed into the decoder to generate an output sequence.
  In other words, the encoder can be thought of as a kind of compression that extracts features from the data. And the decoder can be thought of as playing a role in returning the compressed information in the encoder to its original state.
  So I'm wondering why some models work without having both an encoder and a decoder.</p>
  <p>Few days ago, I think use only encoders are useful to classifying classes. Because <strong>DOSOVITSKIY, Alexey, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.</strong> paper shows only encoder to classification images. Decoders are useful in generative things, because <strong>WANG, Jianfeng, et al. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022.</strong> paper using encoder to encode the visual information from the input image into a representation that can be used by the text decoder to generate text. Then, to generate text, they give the 'encoder's output and the text' as the decoder's input.</p>
  <p>But, I am sure about that my think are wrong because of BERT and GPT. <strong>BERT</strong> using encoder and does not have a decoder. <strong>GPT</strong> uses decoder and does not have a encoder. A typical user thinks that BERT and GPT equally answer the question asked by the user. So they think BERT and GPT provide the same service. However, in terms of model structure, BERT and GPT are completely different.</p>
  <h1>So, I have two questions about each functional part that makes up the transformer.</h1>
  <ol>
  <li>what does encoder and decoder do in transformer? The transformer referred to here can be text or image.</li>
  <li>For what purpose do model designers use only encoders, only decoders, or both encoders and decoders?</li>
  </ol>
  <p>Thank you.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-28T23:49:26.100Z
Comments.Text: I really struggled to get your point, but the whole idea of encoder-decoder, is based on the necessity of conditioning your model... you want just test, just use the  encoder to generate data... you want to condition your text generation with an image, use encoder decoder
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-29T08:37:01.327Z
Comments.Text: @AlbertoSinigaglia I really apologized that you are really struggled to get my point. Now, I edit my question more clearly. If you still hard to get my point, please notice one more time to me. If you notice to me, then I read and edit more clearly and add more explain. Thank you.
Comments.UserDisplayName: Yang
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-04T11:39:56.173Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Tags: <transformer><encoder-decoder><vision-transformer>
Title: Which situation will helpful using encoder or decoder or both in transformer model?
UserDisplayName: Yang
ThreadText: # Which situation will helpful using encoder or decoder or both in transformer model?
  
  (Asked by: username_0 on 2023-07-27)
  
  I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
  
  The overall form of a transformer consists of an encoder and a decoder. Depending on the model, you may use only the encoder, only the decoder, or both. However, for what purpose do model designers use only encoders, only decoders, or both?
  
  I already knew that encoders in transformers are as known as taking in a sequence of input data and generates a fixed-length representation of it. This representation can then be fed into the decoder to generate an output sequence.
  In other words, the encoder can be thought of as a kind of compression that extracts features from the data. And the decoder can be thought of as playing a role in returning the compressed information in the encoder to its original state.
  So I'm wondering why some models work without having both an encoder and a decoder.
  
  Few days ago, I think use only encoders are useful to classifying classes. Because **DOSOVITSKIY, Alexey, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.** paper shows only encoder to classification images. Decoders are useful in generative things, because **WANG, Jianfeng, et al. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022.** paper using encoder to encode the visual information from the input image into a representation that can be used by the text decoder to generate text. Then, to generate text, they give the 'encoder's output and the text' as the decoder's input.
  
  But, I am sure about that my think are wrong because of BERT and GPT. **BERT** using encoder and does not have a decoder. **GPT** uses decoder and does not have a encoder. A typical user thinks that BERT and GPT equally answer the question asked by the user. So they think BERT and GPT provide the same service. However, in terms of model structure, BERT and GPT are completely different.
  
  # So, I have two questions about each functional part that makes up the transformer.
  
  1. what does encoder and decoder do in transformer? The transformer referred to here can be text or image.
  2. For what purpose do model designers use only encoders, only decoders, or both encoders and decoders?
  
  Thank you.
  
  **Tags:** transformer, encoder-decoder, vision-transformer
  
  > **Comments:**
  > **username_1**: I really struggled to get your point, but the whole idea of encoder-decoder, is based on the necessity of conditioning your model... you want just test, just use the  encoder to generate data... you want to condition your text generation with an image, use encoder decoder  
  > **username_0**: @AlbertoSinigaglia I really apologized that you are really struggled to get my point. Now, I edit my question more clearly. If you still hard to get my point, please notice one more time to me. If you notice to me, then I read and edit more clearly and add more explain. Thank you.  
  > **username_2**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-07-29. Score: 10 | Accepted answer
  
  The [original transformer paper](https://arxiv.org/abs/1706.03762) presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a "transformer model", but it actually consists only of an encoder or only of a decoder. That's fine as long as you know what exactly is going on.
  (I would say that the distinctive feature of the transformer model is the [attention layer](https://username_3.github.io/posts/transformer/#attention).)
  
  For example, as you mentioned, **G**enerative **P**retrained **Transformer** (GPT) is a decoder-only model, and **Vi**sion **Transformer** (ViT) is an encoder-only model.
  
  I feel like your question already contains the answer :)
  Yes, encoders are used exactly for that - encoding (or compressing) the input. Note that the output sequence from the encoder has the same length as the input sequence. This is why encoder-only models cannot be used for tasks where the output sequence has a different length or unknown length, e.g. machine translation, text summarization. In these cases you need the decoder.
  
  Encoder-only models like ViT and BERT will encode the input sequence and produce an output sequence of the same length. Then for classification they will use only the first element of the output sequence.
  
  [![Vision Transformer](https://i.sstatic.net/N0vDZ.png)](https://i.sstatic.net/N0vDZ.png)
  
  As I said, encoder-only models cannot be used for generating an output sequence of different (or unknown) length, i.e. BERT cannot be used for machine translation or generative question answering. BERT can only be used for question answering on [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)-type datasets, i.e. your answer is a continuous segment from the input and the model only outputs `<START_IDX>` and `<END_IDX>` to mark it.
  
  For generation tasks you need a decoder. The decoder is actually an auto-regressive model. It will generate the elements of the output sequence one-by-one until it decides that the sequence is ready and then it generates the final `<END>` token. See [here](https://username_3.github.io/posts/transformer/#inference).
  
  Previously people used an encoder-decoder architecture to solve these sequence-to-sequence tasks (e.g. [T5](https://arxiv.org/pdf/1910.10683.pdf)). You want the encoder to encode your input sequence and the decoder to decode it and produce the output sequence. However, it turns out that you can use a decoder-only model. You simply concatenate your source and target sentences and treat the task as a language modelling task. Now there are pros and cons to this approach, but the main takeaway is that you can do it if you want to.
  
  You can checkout these papers to read more about encoder-decoder vs decoder-only models for sequence to sequence tasks:  
  <https://arxiv.org/pdf/2304.04052.pdf>  
  <https://aclanthology.org/2022.aacl-main.43.pdf>
  
  > **Comments:**
  > **username_0**: Thanks for helping about my question. I need more study with your answer. I still confused, because why **google bard**(based on bert, made with only transformer encoder) can do text summmarization on google services. I test google bard with my question, request like "please summarize about given text." and paste my question. The bard gives answer like.... (next comment)  
  > **username_0**: Sure, I can help you with that. Here is the summary of the text you provided:  -Transformer models consist of two main components: an encoder and a decoder. -The encoder takes in a sequence of input data and generates a fixed-length representation of it. -The decoder takes in this representation and generates an output sequence. -In some cases, only the encoder or only the decoder is used. -The encoder is often used for tasks such as classification, while the decoder is often used for tasks such as generation. -However, there are also cases where both the encoder and decoder are used.  
  > **username_0**: And then **bard** give answer about my original question. As we know(according to Pi-tau give knowledge), only encoder model can't do summarization. But google Bard tried to summarize the text. why **bard** looks like **bard** can do summarization? I need more time to search more info, because I want to reask with more background. Thank for helping, and If others still don't write answer on this question, I accept your answer after 1 week. :)  
  > **username_3**: Okay, so first of all Bard is not based on BERT -- It uses LaMDA (https://arxiv.org/pdf/2201.08239.pdf). You can see in the third paragraph of Sec. 3 from the paper that this is a decoder-only model, just like GPT. The difference is that it is pre-trained specifically on dialog data.    >"We use a decoder-only Transformer [ 92 ] language model as the model architecture for LaMDA."  
  > **username_3**: Please note also that Bard is not just a transformer decoder language model. It is a complex system consisting of **many**, **many** components, one of which is the transformer decoder. It is also connected to google's search engine, so it can search the web for information that can help it generate the correct answer. I would recommend this podcast if you want to hear more about how models are trained to use the web: https://www.talkrl.com/episodes/john-schulman.  
  > **username_0**: Thank you! It really help to me, then I understand about my question. :)  
  > **username_4**: So, the difference is about seqlen where encoder is fixed and decoder is dynamic. In that case, encoder also can be used for summarization and generation, just set the fixed seqlen as much as possible (though it's impractical). Where we can also use stop token for both input and target, the dataset is simply the sliding window, where padding token is mandatory here.  
  
  ---
  
  