Answers.Body: <p>Did you mean: </p>
  
  <blockquote>
    <p>How do you use a pre-trained BERT model in a <strong>feature-based</strong> setting
    to get pre-trained <strong>word contextual embeddings</strong>?</p>
  </blockquote>
  
  <p><a href="https://arxiv.org/abs/1810.04805" rel="nofollow noreferrer">Here</a> is the BERT paper. I highly recommend you read it.</p>
  
  <p>Firstly, by <em>sentences</em>, we mean a sequence of <strong>word embedding representations</strong> of the words (or tokens) in the sentence. Word embeddings are the vectors that you mentioned, and so a (usually fixed) sequence of such vectors represent the sentence input. (We don't need the input to always be divided to individual sentences)</p>
  
  <p>There are <strong>mainly</strong> two ways to train the modern neural network models for sequence modeling tasks such as language modelling, machine translation and question answering:</p>
  
  <h2>1. The pretrain - finetune based approach (also called Transfer Learning)</h2>
  
  <p>In this approach, a model is first pre-trained on some auxiliary dataset whose domain usually overlaps with the domain of the dataset corresponding to the <strong>downstream task</strong>, that is, the actual task for which we need to build the model.</p>
  
  <p>In the case of Natural Language Processing (NLP) tasks, such as machine translation and question answering, the pre-training that is done is usually unsupervised (no labels in the dataset) where our training objective is to maximize the <strong>log likelihood</strong> of the sentences. You can read more about Maximum Likelihood estimation <a href="http://pages.cs.wisc.edu/~jerryzhu/cs838/LM.pdf" rel="nofollow noreferrer">here</a>.</p>
  
  <p>Its almost always a good idea to pre-train models (whether it is for NLP tasks or for computer vision tasks etc). The more we pre-train a model, the closer its parameters are to the optimal ones, and the work needed in the <strong>fine-tuning</strong> phase is minimized. This is why we use large scale datasets for pre-training.</p>
  
  <p>Later, during the fine-tuning phase, we fine tune the parameters of the model to suit a downstream task, such as a task of translation between two <strong>specific</strong> languages of text. Since the parameters of the model are already at a position in the error/loss space that is already <strong>sort of good enough</strong> for the model (because we already pre-trained it with text in, say, 10 languages), the fine-tuning can, in a sense, tailor the model to suit the task a hand.</p>
  
  <h2>2. The feature based approach</h2>
  
  <p>In this approach, we take an already pre-trained model (any model, e.g. a transformer based neural net such as BERT, which has been <strong>pre-trained</strong> as described in the previous approach) and then we extract the activations from one or more layers of this pre-trained model. </p>
  
  <p>This is done by simply inputting the word embedding sequence corresponding to a sentence to the pre-trained model and then extracting the activations from one or more layers (one or more of the last layers, since the features associated to the activations in these layers are far more complex and include more context) of this pre-trained model. </p>
  
  <p>These activations (also called <strong>contextual embeddings</strong>) are used as input (similar to the word embeddings described earlier) to another model, such as an LSTM or even another BERT. That is, fixed features are extracted from the pre-trained model.</p>
  
  <p>Usually the size of these layers are the same as the size of the input layer, i.e. they are also a sequence, but a sequence of far more complex features instead of word embeddings. If the word embedding vector of the input to this model id <span class="math-container">$D$</span> and the length of the sentence which is input to the model is <span class="math-container">$L$</span>, then each of the hidden activation layers have the same length <span class="math-container">$L$</span>, but the hidden size of these <strong>vectors</strong> is <span class="math-container">$H &gt;&gt; D$</span>, i.e. far higher to incorporate more complex features.</p>
  
  <p>Even though the dimensionality of the hidden layers is far higher, these layers are equal in length to the input layer, and so we have a one to one correspondence between the input tokens in the sentence and each of these hidden layer activations. </p>
  
  <p>The bonus in using these hidden representations is that we get to use representations of the input sentence that also incorporate <strong>CONTEXT</strong>, dependency between the words of the sentence, an important feature that is to be modeled by all models used for NLP tasks. Models such as BERT, are based on Transformer models that use an attention mechanism in order to model these contextual features.</p>
  
  <p><a href="https://i.sstatic.net/Wu3Q4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Wu3Q4.png" alt="enter image description here"></a></p>
  
  <p><a href="http://jalammar.github.io/illustrated-transformer/" rel="nofollow noreferrer">This</a> is a superb article describing to the last detail the attention mechanism and the Transformer model based on which models like BERT operate. For more about contextual embeddings I recommend taking a look at <a href="http://jalammar.github.io/illustrated-bert/" rel="nofollow noreferrer">this</a> article. The figure used here is from this article.</p>
  
  <p>Models such as BERT make use of <strong>one half</strong> of these Transformer models, called the <strong>encoder</strong>, since the Transformer is a <strong>seq2seq</strong> or <strong>encoder-decoder model</strong>. <a href="https://arxiv.org/abs/1706.03762" rel="nofollow noreferrer">Here</a> is the link to the Transformer paper.</p>
  
  <p><a href="https://towardsdatascience.com/nlp-extract-contextualized-word-embeddings-from-bert-keras-tf-67ef29f60a7b" rel="nofollow noreferrer">Here</a> is a great blog on extracting contextual word embeddings from BERT using Tensorflow and Keras.</p>
  
  <p>I'll also provide a link to a <a href="https://www.kaggle.com/funtowiczmo/hugging-face-transformers-how-to-use-pipelines" rel="nofollow noreferrer">Kaggle Python Notebook</a> on using Pipelines functionality from the HuggingFace community repo on github that also is used for feature extraction (contextual embeddings).</p>
  
  <p>Please let me know if you have any questions, happy to help!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T18:59:50.167Z
Answers.LastActivityDate: 2020-05-11T21:08:28.290Z
Answers.LastEditDate: 2020-05-11T21:08:28.290Z
Answers.UserDisplayName: samirzach
Body: <p>I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing.</p>
  <blockquote>
  <p><strong>Is it possible to get the vectors using pre-trained BERT from the data?</strong></p>
  </blockquote>
  
Tags: <natural-language-processing><bert><pretrained-models>
Title: How to use pre-trained BERT to extract the vectors from sentences?
UserDisplayName: Pluviophile
ThreadText: # How to use pre-trained BERT to extract the vectors from sentences?
  
  (Asked by: username_0 on 2020-04-27)
  
  I'm trying to extract the vectors from the sentences. Spent soo much time searching for the pre-trained BERT models but found nothing.
  
  > **Is it possible to get the vectors using pre-trained BERT from the data?**
  
  **Tags:** natural-language-processing, bert, pretrained-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-11. Score: 4 | Accepted answer
  
  Did you mean:
  
  > How do you use a pre-trained BERT model in a **feature-based** setting
  > to get pre-trained **word contextual embeddings**?
  
  [Here](https://arxiv.org/abs/1810.04805) is the BERT paper. I highly recommend you read it.
  
  Firstly, by *sentences*, we mean a sequence of **word embedding representations** of the words (or tokens) in the sentence. Word embeddings are the vectors that you mentioned, and so a (usually fixed) sequence of such vectors represent the sentence input. (We don't need the input to always be divided to individual sentences)
  
  There are **mainly** two ways to train the modern neural network models for sequence modeling tasks such as language modelling, machine translation and question answering:
  
  ## 1. The pretrain - finetune based approach (also called Transfer Learning)
  
  In this approach, a model is first pre-trained on some auxiliary dataset whose domain usually overlaps with the domain of the dataset corresponding to the **downstream task**, that is, the actual task for which we need to build the model.
  
  In the case of Natural Language Processing (NLP) tasks, such as machine translation and question answering, the pre-training that is done is usually unsupervised (no labels in the dataset) where our training objective is to maximize the **log likelihood** of the sentences. You can read more about Maximum Likelihood estimation [here](http://pages.cs.wisc.edu/~jerryzhu/cs838/LM.pdf).
  
  Its almost always a good idea to pre-train models (whether it is for NLP tasks or for computer vision tasks etc). The more we pre-train a model, the closer its parameters are to the optimal ones, and the work needed in the **fine-tuning** phase is minimized. This is why we use large scale datasets for pre-training.
  
  Later, during the fine-tuning phase, we fine tune the parameters of the model to suit a downstream task, such as a task of translation between two **specific** languages of text. Since the parameters of the model are already at a position in the error/loss space that is already **sort of good enough** for the model (because we already pre-trained it with text in, say, 10 languages), the fine-tuning can, in a sense, tailor the model to suit the task a hand.
  
  ## 2. The feature based approach
  
  In this approach, we take an already pre-trained model (any model, e.g. a transformer based neural net such as BERT, which has been **pre-trained** as described in the previous approach) and then we extract the activations from one or more layers of this pre-trained model.
  
  This is done by simply inputting the word embedding sequence corresponding to a sentence to the pre-trained model and then extracting the activations from one or more layers (one or more of the last layers, since the features associated to the activations in these layers are far more complex and include more context) of this pre-trained model.
  
  These activations (also called **contextual embeddings**) are used as input (similar to the word embeddings described earlier) to another model, such as an LSTM or even another BERT. That is, fixed features are extracted from the pre-trained model.
  
  Usually the size of these layers are the same as the size of the input layer, i.e. they are also a sequence, but a sequence of far more complex features instead of word embeddings. If the word embedding vector of the input to this model id $D$ and the length of the sentence which is input to the model is $L$, then each of the hidden activation layers have the same length $L$, but the hidden size of these **vectors** is $H >> D$, i.e. far higher to incorporate more complex features.
  
  Even though the dimensionality of the hidden layers is far higher, these layers are equal in length to the input layer, and so we have a one to one correspondence between the input tokens in the sentence and each of these hidden layer activations.
  
  The bonus in using these hidden representations is that we get to use representations of the input sentence that also incorporate **CONTEXT**, dependency between the words of the sentence, an important feature that is to be modeled by all models used for NLP tasks. Models such as BERT, are based on Transformer models that use an attention mechanism in order to model these contextual features.
  
  [![enter image description here](https://i.sstatic.net/Wu3Q4.png)](https://i.sstatic.net/Wu3Q4.png)
  
  [This](http://jalammar.github.io/illustrated-transformer/) is a superb article describing to the last detail the attention mechanism and the Transformer model based on which models like BERT operate. For more about contextual embeddings I recommend taking a look at [this](http://jalammar.github.io/illustrated-bert/) article. The figure used here is from this article.
  
  Models such as BERT make use of **one half** of these Transformer models, called the **encoder**, since the Transformer is a **seq2seq** or **encoder-decoder model**. [Here](https://arxiv.org/abs/1706.03762) is the link to the Transformer paper.
  
  [Here](https://towardsdatascience.com/nlp-extract-contextualized-word-embeddings-from-bert-keras-tf-67ef29f60a7b) is a great blog on extracting contextual word embeddings from BERT using Tensorflow and Keras.
  
  I'll also provide a link to a [Kaggle Python Notebook](https://www.kaggle.com/funtowiczmo/hugging-face-transformers-how-to-use-pipelines) on using Pipelines functionality from the HuggingFace community repo on github that also is used for feature extraction (contextual embeddings).
  
  Please let me know if you have any questions, happy to help!
  
  
  ---
  
  