Answers.Body: <p>If you're eventually building a classifier with added features, concatenating the LM output embeddings with those additional features should work, I believe.</p>
  <p>It seems similar to incorporating non-sequential data, like location, in RNN models, where the final set of features for classification are last hidden layer output + non-sequential features.</p>
  <p><a href="https://i.sstatic.net/PwlSH.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PwlSH.jpg" alt="incorporating non-sequential data" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-21T18:48:00.357Z
Answers.LastActivityDate: 2022-10-21T18:48:00.357Z
Answers.UserDisplayName: Ashwani Yadav
Answers.Body: <p>Although barely related, this brings to my mind a concept I first learned in <em>conditional GANs</em> - there, if you wanted an output of some specific class, you would take your random noise input <span class="math-container">$z$</span> (that is given to your generative model <span class="math-container">$G$</span>) and concatenate it with some representation of the class instance. On the <code>MNIST</code> digits dataset, for example, you can provide <span class="math-container">$[z,(1,0,...,0)]$</span> where <span class="math-container">$(1,0,...,0)$</span> represents a one-hot encoding of the digit <span class="math-container">$0$</span>. So <em>perhaps</em> you could use this concatenation concept for your task:</p>
  <ol>
  <li>extract features <span class="math-container">$\{a_i\}_{i=1}^N$</span> from the text itself - this can be done using a transformer, as suggested, or some RNN architecture like LSTM or GRU.</li>
  <li>extract features <span class="math-container">$\{b_i\}_{i=1}^N$</span> from the metadata - there are some approaches that deal with tabular data, but I would start with a basic MLP</li>
  <li>concatenate the two feature vectors <span class="math-container">$c_i=[a_i,b_i]$</span></li>
  <li>yield a final prediction using yet another model that takes <span class="math-container">$\{c_i\}_{i=1}^N$</span> as input</li>
  </ol>
  <hr />
  <p>Edit: I've looked this up online, and found some supporting evidence - check out <a href="https://towardsdatascience.com/integrating-image-and-tabular-data-for-deep-learning-9281397c7318" rel="nofollow noreferrer">this</a> blog post, for example.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-10T08:20:08.947Z
Answers.LastActivityDate: 2022-09-10T08:20:08.947Z
Answers.UserDisplayName: Hadar Sharvit
Body: <p>Are there any techniques to combine a feature set (other than the text itself) with pretrained language models.</p>
  <p>Let's say I have a random NLP task that tries to predict a binary class label based on e.g. Twitter data. One could easily utilize a pretrained language model such as BERT/GPT-3 etc. to fine-tune it on the text of the tweets. However the tweets come with a lot of useful metadata such as likes/retweets etc. or if I want to add additional syntactic features such as POS-Tags, dependency relation or any other generated feature. Is it possible to use additional features I extracted for the finetuning step of the pretrained language model? Or is the only way of doing so to use an ensemble classifier and basically write a classifier for each of the extracted features and combine all of their predictions with the finetuned LMs predictions?</p>
  
Tags: <natural-language-processing><bert><fine-tuning><language-model>
Title: How to combine pretrained language models with additional feature set?
UserDisplayName: fragant
ThreadText: # How to combine pretrained language models with additional feature set?
  
  (Asked by: username_0 on 2022-09-08)
  
  Are there any techniques to combine a feature set (other than the text itself) with pretrained language models.
  
  Let's say I have a random NLP task that tries to predict a binary class label based on e.g. Twitter data. One could easily utilize a pretrained language model such as BERT/GPT-3 etc. to fine-tune it on the text of the tweets. However the tweets come with a lot of useful metadata such as likes/retweets etc. or if I want to add additional syntactic features such as POS-Tags, dependency relation or any other generated feature. Is it possible to use additional features I extracted for the finetuning step of the pretrained language model? Or is the only way of doing so to use an ensemble classifier and basically write a classifier for each of the extracted features and combine all of their predictions with the finetuned LMs predictions?
  
  **Tags:** natural-language-processing, bert, fine-tuning, language-model
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-10-21. Score: 1
  
  If you're eventually building a classifier with added features, concatenating the LM output embeddings with those additional features should work, I believe.
  
  It seems similar to incorporating non-sequential data, like location, in RNN models, where the final set of features for classification are last hidden layer output + non-sequential features.
  
  [![incorporating non-sequential data](https://i.sstatic.net/PwlSH.jpg)](https://i.sstatic.net/PwlSH.jpg)
  
  
  ---
  
  ### Answer by username_2 on 2022-09-10. Score: 1
  
  Although barely related, this brings to my mind a concept I first learned in *conditional GANs* - there, if you wanted an output of some specific class, you would take your random noise input $z$ (that is given to your generative model $G$) and concatenate it with some representation of the class instance. On the `MNIST` digits dataset, for example, you can provide $[z,(1,0,...,0)]$ where $(1,0,...,0)$ represents a one-hot encoding of the digit $0$. So *perhaps* you could use this concatenation concept for your task:
  
  1. extract features $\{a\_i\}\_{i=1}^N$ from the text itself - this can be done using a transformer, as suggested, or some RNN architecture like LSTM or GRU.
  2. extract features $\{b\_i\}\_{i=1}^N$ from the metadata - there are some approaches that deal with tabular data, but I would start with a basic MLP
  3. concatenate the two feature vectors $c\_i=[a\_i,b\_i]$
  4. yield a final prediction using yet another model that takes $\{c\_i\}\_{i=1}^N$ as input
  
  ---
  
  Edit: I've looked this up online, and found some supporting evidence - check out [this](https://towardsdatascience.com/integrating-image-and-tabular-data-for-deep-learning-9281397c7318) blog post, for example.
  
  
  ---
  
  