Answers.Body: <p><a href="https://i.sstatic.net/Ahjz1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Ahjz1.png" alt="enter image description here"></a></p>
  
  <p>Source: <a href="https://blog.easysol.net/building-ai-applications/" rel="nofollow noreferrer">https://blog.easysol.net/building-ai-applications/</a></p>
  
  <p>When data is too big, complex and nonlinear it's time to try deep learning. It's always good to try to add some layers to see it can eliminate bias and don't lead to high variance. </p>
  
  <p>Deep learning models can be tweaked(hyperparameters) and regularized(parameters), and it is worth the work.  </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T14:37:23.887Z
Answers.LastActivityDate: 2020-02-21T14:37:23.887Z
Answers.UserDisplayName: Lerner Zhang
Body: <p>I did an out of domain detection task (as a binary classification problem) and tried LR and Naive Bayes and BERT but the deep neural network didn't perform better than LR and NB. For the LR I just used BOW and it beats the 12-layer BERT. </p>
  
  <p>In a lecture, <a href="https://www.youtube.com/watch?v=HfM8UIohGE0" rel="nofollow noreferrer">Andrew Ng</a> suggests "Build First System Quickly, Then Iterate", but it turns out that sometimes we don't need to iterate the model into a deep neural network and most of the time traditional shallow neural networks are good/competitive enough and much simpler for training. </p>
  
  <p>As <a href="https://twitter.com/dirk_hovy/status/1222902471246278657" rel="nofollow noreferrer">this tweet (and its replies)</a> indicate, together with various papers [<a href="https://arxiv.org/pdf/1503.00693.pdf" rel="nofollow noreferrer">1</a>, <a href="https://www.aclweb.org/anthology/S18-1003.pdf" rel="nofollow noreferrer">2</a>, <a href="https://www.aclweb.org/anthology/S18-1004.pdf" rel="nofollow noreferrer">3</a>, <a href="https://paperswithcode.com/sota/sentiment-analysis-on-imdb" rel="nofollow noreferrer">4</a> etc], traditional SVM, LR, and Naive Bayes can beat RNN and some complicated neural networks. </p>
  
  <p>Then my two questions are: </p>
  
  <ol>
  <li><p>When should we switch to complicated neural networks like RNN, CNN, and transformer and etc? How can we see that from the data set or the results (by doing error analysis) of the simple neural networks?       </p></li>
  <li><p>The aforementioned experiments may be caused by the simple test set, then (how) is it possible for us to design a test set that can fail the traditional models? </p></li>
  </ol>
  
Tags: <natural-language-processing><classification><text-classification>
Title: When is it time to switch to deep neural networks from simple networks in text classification problems?
UserDisplayName: Lerner Zhang
ThreadText: # When is it time to switch to deep neural networks from simple networks in text classification problems?
  
  (Asked by: username_0 on 2020-02-18)
  
  I did an out of domain detection task (as a binary classification problem) and tried LR and Naive Bayes and BERT but the deep neural network didn't perform better than LR and NB. For the LR I just used BOW and it beats the 12-layer BERT.
  
  In a lecture, [Andrew Ng](https://www.youtube.com/watch?v=HfM8UIohGE0) suggests "Build First System Quickly, Then Iterate", but it turns out that sometimes we don't need to iterate the model into a deep neural network and most of the time traditional shallow neural networks are good/competitive enough and much simpler for training.
  
  As [this tweet (and its replies)](https://twitter.com/dirk_hovy/status/1222902471246278657) indicate, together with various papers [[1](https://arxiv.org/pdf/1503.00693.pdf), [2](https://www.aclweb.org/anthology/S18-1003.pdf), [3](https://www.aclweb.org/anthology/S18-1004.pdf), [4](https://paperswithcode.com/sota/sentiment-analysis-on-imdb) etc], traditional SVM, LR, and Naive Bayes can beat RNN and some complicated neural networks.
  
  Then my two questions are:
  
  1. When should we switch to complicated neural networks like RNN, CNN, and transformer and etc? How can we see that from the data set or the results (by doing error analysis) of the simple neural networks?
  2. The aforementioned experiments may be caused by the simple test set, then (how) is it possible for us to design a test set that can fail the traditional models?
  
  **Tags:** natural-language-processing, classification, text-classification
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-02-21. Score: 1
  
  [![enter image description here](https://i.sstatic.net/Ahjz1.png)](https://i.sstatic.net/Ahjz1.png)
  
  Source: <https://blog.easysol.net/building-ai-applications/>
  
  When data is too big, complex and nonlinear it's time to try deep learning. It's always good to try to add some layers to see it can eliminate bias and don't lead to high variance.
  
  Deep learning models can be tweaked(hyperparameters) and regularized(parameters), and it is worth the work.
  
  
  ---
  
  