Answers.Body: <p>I think you are thinking too much like traditional programming here. A neural network doesn't need if-else. Providing the special tokens hint to the model about the structure of the question, and with enough data and training, the language model learns to utilize that hint.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-15T04:48:43.897Z
Answers.LastActivityDate: 2022-07-15T04:48:43.897Z
Answers.UserDisplayName: DKDK
Body: <p>&quot;[SEP] tokens are useful to differentiate the questions from answers through type_ids&quot; Yes, but how is this helping model to understand that &quot;I should look paragraph and generate answers from here&quot;? We don't have if-else inside the model that will say: &quot;if type_id==1, generate questions from here&quot;</p>
  <p>The same question appears for this example too:</p>
  <p>[CLS] previous question [Q] current question [SEP] text [EOS]</p>
  <p>How model says: &quot;I should look at the previous question, understand the meaning from here too with the current question, and answer the question.&quot; We need if-else in here too like:</p>
  <p>If there is a previous question:
  get the meaning from it and use it with the current question</p>
  <p>One more example based on this paper:<a href="https://arxiv.org/pdf/2005.01107v1.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2005.01107v1.pdf</a>
  In this paper, we have a dataset like this: <a href="https://huggingface.co/datasets/iarfmoose/question_generator" rel="nofollow noreferrer">https://huggingface.co/datasets/iarfmoose/question_generator</a></p>
  <p>In the t5 transformer, if we don't have those  and  tokens, the model will not learn anything. And I have no idea how specifying  and  we are helping the model to generate questions based on that answer in the context.</p>
  <p>I hope I am clear with my question.</p>
  
Tags: <neural-networks><deep-learning><natural-language-processing><transformer>
Title: How special tokens in BERT-Transformers work?
UserDisplayName: canP
ThreadText: # How special tokens in BERT-Transformers work?
  
  (Asked by: username_0 on 2022-07-14)
  
  "[SEP] tokens are useful to differentiate the questions from answers through type\_ids" Yes, but how is this helping model to understand that "I should look paragraph and generate answers from here"? We don't have if-else inside the model that will say: "if type\_id==1, generate questions from here"
  
  The same question appears for this example too:
  
  [CLS] previous question [Q] current question [SEP] text [EOS]
  
  How model says: "I should look at the previous question, understand the meaning from here too with the current question, and answer the question." We need if-else in here too like:
  
  If there is a previous question:
  get the meaning from it and use it with the current question
  
  One more example based on this paper:<https://arxiv.org/pdf/2005.01107v1.pdf>
  In this paper, we have a dataset like this: <https://huggingface.co/datasets/iarfmoose/question_generator>
  
  In the t5 transformer, if we don't have those and tokens, the model will not learn anything. And I have no idea how specifying and we are helping the model to generate questions based on that answer in the context.
  
  I hope I am clear with my question.
  
  **Tags:** neural-networks, deep-learning, natural-language-processing, transformer
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-15. Score: 2
  
  I think you are thinking too much like traditional programming here. A neural network doesn't need if-else. Providing the special tokens hint to the model about the structure of the question, and with enough data and training, the language model learns to utilize that hint.
  
  
  ---
  
  