Answers.Body: <p>BERT is pre-trained using MLM and NSP tasks:</p>
  <ul>
  <li>The MLM task is a type of self-supervised learning where some of the words in a sentence are randomly masked, and the model is trained to predict the masked words based on the surrounding context. For example, in the sentence &quot;The cat is on the mat&quot;, a masked version could be &quot;The [MASK] is on the mat&quot;, and the model is trained to predict the missing word (in this case, &quot;cat&quot;).</li>
  <li>The goal of the NSP task is to train BERT to understand the relationships between sentences. It helps BERT learn not just about individual words, but also about how sentences are connected in a broader context. The loss function used for the NSP task is a standard binary cross-entropy loss. It measures the difference between the predicted probability of &quot;IsNext&quot; and the actual label.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-24T14:37:49.893Z
Answers.LastActivityDate: 2023-10-24T14:37:49.893Z
Answers.UserDisplayName: Larsen
Answers.Body: <p>I found the following in the original BERT paper <a href="https://arxiv.org/pdf/1810.04805v2.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1810.04805v2.pdf</a> : &quot;The
  training loss is the sum of the mean masked LM [language model]
  likelihood and the mean next sentence prediction
  likelihood.&quot;</p>
  <hr />
  <p>The original BERT model was trained to perform two natural language processing tasks: masked language modeling and next sentence prediction.</p>
  <p>In masked language modeling, the model is given a sentence with some of the words masked out, and its task is to predict the original value of the masked words, based on the context provided by the rest of the sentence. This task is useful for learning contextual representations of words, because the model has to consider the full context in which the word appears in order to make a prediction.</p>
  <p>In next sentence prediction, the model is given a pair of sentences and has to predict whether the second sentence is likely to follow the first one in a coherent text. This task helps the model learn about the relationships between sentences and how they fit together in a larger text.</p>
  <p>The original BERT model was trained using a combination of these two tasks, with the masked language modeling task being the primary task and the next sentence prediction task being used as an auxiliary task. The model was trained to minimize the cross-entropy loss on these tasks, which is a common choice for classification tasks like these.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-07T09:57:19.310Z
Answers.LastActivityDate: 2022-12-30T18:18:31.650Z
Answers.LastEditDate: 2022-12-30T18:18:31.650Z
Answers.UserDisplayName: Ggjj11
Body: <p>I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.</p>
  <p>I understand that BERT applies a WordPice tokenizer (e.g. working like <a href="https://keras.io/api/keras_nlp/tokenizers/word_piece_tokenizer/" rel="nofollow noreferrer">https://keras.io/api/keras_nlp/tokenizers/word_piece_tokenizer/</a>) and then passes the tokens through several (transformer) layers.
  If using the transformers library, the output of each hidden layer can be accessed easily as described here  <a href="https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/" rel="nofollow noreferrer">https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/</a> .
  For each token we can then obtain a word embedding and aggregate a sentence embedding by e.g. mean- or max-pooling over all word embeddings in a sentence.</p>
  <p>On <a href="https://d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html" rel="nofollow noreferrer">https://d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html</a> , I found that BERT can be trained on e.g. WikiText-2 (<a href="https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/" rel="nofollow noreferrer">https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/</a> ) but I do not see</p>
  <ol>
  <li>on which training task</li>
  <li>which loss function the original BERT model is trained?</li>
  </ol>
  <p>This is curcial, since it determines what pattern the model picks up.</p>
  <p>The last website states that the loss function is a cross-entropy loss. But I do not yet understand corss-entropy between what? What is the (X,y)-pairs used for training Bert?</p>
  
Tags: <bert>
Title: What is the loss function and training task on which the original BERT model was trained
UserDisplayName: Ggjj11
ThreadText: # What is the loss function and training task on which the original BERT model was trained
  
  (Asked by: username_0 on 2022-09-07)
  
  I was checking on sentence embeddings and stumbled across the BERT model which employs transformers.
  
  I understand that BERT applies a WordPice tokenizer (e.g. working like <https://keras.io/api/keras_nlp/tokenizers/word_piece_tokenizer/>) and then passes the tokens through several (transformer) layers.
  If using the transformers library, the output of each hidden layer can be accessed easily as described here <https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/> .
  For each token we can then obtain a word embedding and aggregate a sentence embedding by e.g. mean- or max-pooling over all word embeddings in a sentence.
  
  On <https://d2l.ai/chapter_natural-language-processing-pretraining/bert-pretraining.html> , I found that BERT can be trained on e.g. WikiText-2 (<https://blog.salesforceairesearch.com/the-wikitext-long-term-dependency-language-modeling-dataset/> ) but I do not see
  
  1. on which training task
  2. which loss function the original BERT model is trained?
  
  This is curcial, since it determines what pattern the model picks up.
  
  The last website states that the loss function is a cross-entropy loss. But I do not yet understand corss-entropy between what? What is the (X,y)-pairs used for training Bert?
  
  **Tags:** bert
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2022-09-07. Score: 3 | Accepted answer
  
  I found the following in the original BERT paper <https://arxiv.org/pdf/1810.04805v2.pdf> : "The
  training loss is the sum of the mean masked LM [language model]
  likelihood and the mean next sentence prediction
  likelihood."
  
  ---
  
  The original BERT model was trained to perform two natural language processing tasks: masked language modeling and next sentence prediction.
  
  In masked language modeling, the model is given a sentence with some of the words masked out, and its task is to predict the original value of the masked words, based on the context provided by the rest of the sentence. This task is useful for learning contextual representations of words, because the model has to consider the full context in which the word appears in order to make a prediction.
  
  In next sentence prediction, the model is given a pair of sentences and has to predict whether the second sentence is likely to follow the first one in a coherent text. This task helps the model learn about the relationships between sentences and how they fit together in a larger text.
  
  The original BERT model was trained using a combination of these two tasks, with the masked language modeling task being the primary task and the next sentence prediction task being used as an auxiliary task. The model was trained to minimize the cross-entropy loss on these tasks, which is a common choice for classification tasks like these.
  
  
  ---
  
  ### Answer by username_1 on 2023-10-24. Score: 0
  
  BERT is pre-trained using MLM and NSP tasks:
  
  * The MLM task is a type of self-supervised learning where some of the words in a sentence are randomly masked, and the model is trained to predict the masked words based on the surrounding context. For example, in the sentence "The cat is on the mat", a masked version could be "The [MASK] is on the mat", and the model is trained to predict the missing word (in this case, "cat").
  * The goal of the NSP task is to train BERT to understand the relationships between sentences. It helps BERT learn not just about individual words, but also about how sentences are connected in a broader context. The loss function used for the NSP task is a standard binary cross-entropy loss. It measures the difference between the predicted probability of "IsNext" and the actual label.
  
  
  ---
  
  