Body: <p>I have recently come across transformers, I am new to Deep Learning. I have seen a <a href="https://arxiv.org/pdf/2007.10819.pdf" rel="nofollow noreferrer">paper</a> using   CNN and BiLSTM on top of a transformer, the paper uses a transformer(XLM-R) for sentiment analysis in code-mixed domain. But many of the blogs only use a normal feed formal network on top of the transformer.</p>
  <p>I am trying to use transformers for sentiment analysis, short text classification.</p>
  <p>Is it overkill to use models like CNN and BiLSTM on top of the transformer considering the size of the data it is trained on and its complexity?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T03:53:15.950Z
Comments.Text: Hello. Welcome to AI SE. It would be better if you describe the problem you're trying to solve with the transformer and LSTM and what that paper is using them for.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T04:34:56.647Z
Comments.Text: The paper is using transformers for sentiment analysis on code mixed domain
Comments.UserDisplayName: learner
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-25T12:42:16.117Z
Comments.Text: It may be useful to also know what those blog posts that you mention use the LSTM + transformer for. Do they use it for the same task? Maybe provide a link to one such blog post.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-26T04:54:36.223Z
Comments.Text: https://medium.com/atheros/text-classification-with-transformers-in-tensorflow-2-bert-2f4f16eff5ad
Comments.UserDisplayName: learner
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-26T04:56:22.950Z
Comments.Text: The hugging face implementation of transformers for sequence classification uses a feed-forward layer on top of the transformer
Comments.UserDisplayName: learner
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-02T13:00:35.363Z
Comments.Text: How would using a LSTM after a transformer improve the performance? I mean, the transformers does all the magic a LSTM does (it even sees the whole context at the same time). A pretrained transformer + a linear layer is the normal way to do it, to the best of my knowledge..
Comments.UserDisplayName: Isbister
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-05T10:14:01.590Z
Comments.Text: Using transformer as a feature extractor for the LSTM
Comments.UserDisplayName: learner
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-16T12:14:13.490Z
Comments.Text: But all the magic a LSTM does is captured by attention. So when you finetune on your target task the properties of memory (that lstms provide) are already captured by the transformer.
Comments.UserDisplayName: Isbister
Tags: <deep-learning><natural-language-processing><transformer><text-classification><sentiment-analysis>
Title: Is using a LSTM, CNN or any other neural network model on top of a Transformer(using hidden states) overkill?
UserDisplayName: learner
ThreadText: # Is using a LSTM, CNN or any other neural network model on top of a Transformer(using hidden states) overkill?
  
  (Asked by: username_0 on 2021-01-25)
  
  I have recently come across transformers, I am new to Deep Learning. I have seen a [paper](https://arxiv.org/pdf/2007.10819.pdf) using CNN and BiLSTM on top of a transformer, the paper uses a transformer(XLM-R) for sentiment analysis in code-mixed domain. But many of the blogs only use a normal feed formal network on top of the transformer.
  
  I am trying to use transformers for sentiment analysis, short text classification.
  
  Is it overkill to use models like CNN and BiLSTM on top of the transformer considering the size of the data it is trained on and its complexity?
  
  **Tags:** deep-learning, natural-language-processing, transformer, text-classification, sentiment-analysis
  
  > **Comments:**
  > **username_1**: Hello. Welcome to AI SE. It would be better if you describe the problem you're trying to solve with the transformer and LSTM and what that paper is using them for.  
  > **username_0**: The paper is using transformers for sentiment analysis on code mixed domain  
  > **username_1**: It may be useful to also know what those blog posts that you mention use the LSTM + transformer for. Do they use it for the same task? Maybe provide a link to one such blog post.  
  > **username_0**: https://medium.com/atheros/text-classification-with-transformers-in-tensorflow-2-bert-2f4f16eff5ad  
  > **username_0**: The hugging face implementation of transformers for sequence classification uses a feed-forward layer on top of the transformer  
  > **username_2**: How would using a LSTM after a transformer improve the performance? I mean, the transformers does all the magic a LSTM does (it even sees the whole context at the same time). A pretrained transformer + a linear layer is the normal way to do it, to the best of my knowledge..  
  > **username_0**: Using transformer as a feature extractor for the LSTM  
  > **username_2**: But all the magic a LSTM does is captured by attention. So when you finetune on your target task the properties of memory (that lstms provide) are already captured by the transformer.  
  
  ---
  
  ## 0 answers
  
  