Answers.Body: <p>Based on my experience, I did 2 tasks that is proven to improve the accuracy/score of my model.</p>
  
  <ol>
  <li>Normalization
  
  <ul>
  <li>removing characters and symbols in a text </li>
  <li>lowercase folding</li>
  </ul></li>
  <li>Stopwords removal (as what you asked)</li>
  </ol>
  
  <p>These process helped me improve my model since stopwords gave my model noise as I am using word frequency count to represent text. </p>
  
  <p>So based on what you asked, does stopwords removal improve score? It depends on your model. If you are using word count to represent text you may do stopwords removal to remove noise when doing text classification.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T09:31:14.507Z
Answers.LastActivityDate: 2020-04-21T09:31:14.507Z
Answers.UserDisplayName: Angela Marpaung
Answers.Body: <p>Based on my project, there is how i clean and doing some preparation on the data.</p>
  
  <ol>
  <li>Delete specific charaters ('\r', '\n', '"',) </li>
  <li>Change into the lowercase</li>
  <li>Delete some symbols</li>
  <li>Lemmatization (change base word with wordnet)</li>
  <li>Delete stopwords.</li>
  </ol>
  
  <p>With these following step, i get some improvement accuracy score on my model.</p>
  
  <p>My project: <a href="https://github.com/khaifagifari/NLP-Course-TelU" rel="nofollow noreferrer">https://github.com/khaifagifari/NLP-Course-TelU</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T08:46:51.947Z
Answers.LastActivityDate: 2020-04-21T08:46:51.947Z
Answers.UserDisplayName: Muhammad Khaifa Gifari
Answers.Body: <p>The technical term for these words is "<a href="https://en.wikipedia.org/wiki/Stop_words" rel="nofollow noreferrer">stop words</a>". Have a look at <a href="https://en.wikipedia.org/wiki/Information_retrieval" rel="nofollow noreferrer">Information Retrieval</a> and indexing (eg <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf" rel="nofollow noreferrer">TF/IDF</a>) to make up your mind whether you want to remove them or not.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-16T16:34:56.957Z
Answers.LastActivityDate: 2020-04-16T16:34:56.957Z
Answers.UserDisplayName: Oliver Mason
Answers.Body: <p>As far as I know, there are few <strong>aspects</strong> that would probably improve the model score:</p>
  
  <blockquote>
    <ol>
    <li>Normalization</li>
    <li>Lemmatization</li>
    <li>Stopwords removal (as you asked here)</li>
    </ol>
  </blockquote>
  
  <p>Based on your question, "is removing top frequent words (stopwords) will improve the model score?". The answer is, it depends on what kind of stopwords are you removing. The problem here is that if you do not remove stop words, the noise will increase in the dataset because of words like I, my, me, etc. Here is the comparison of those three <strong>aspects</strong> using <strong>SVM Classifier</strong>.</p>
  
  <p><a href="https://i.sstatic.net/BQavF.png" rel="noreferrer"><img src="https://i.sstatic.net/BQavF.png" alt="Comparison using SVM Classifier"></a></p>
  
  <p>You may see that without stopwords removal the Train Set Accuracy decreased to <strong>94.81%</strong> and the Test Set Accuracy decreased to <strong>88.02%</strong>. But, you should be careful about what kind of stopwords you are removing. </p>
  
  <p>If you are working with basic NLP techniques like BOW, Count Vectorizer or TF-IDF(Term Frequency and Inverse Document Frequency) then removing stopwords is a good idea because stopwords act like noise for these methods. If you working with LSTM’s or other models which capture the semantic meaning and the meaning of a word depends on the context of the previous text, then it becomes important not to remove stopwords. </p>
  
  <p>So, what's the solution?</p>
  
  <blockquote>
    <p>You may want to create a Python package <strong>nlppreprocess</strong> which removes stops words that are not necessary. It also has some additional functionalities that can make cleaning of text fast. For example:</p>
  </blockquote>
  
  <pre><code>from nlppreprocess import NLP
  import pandas as pd
  
  nlp = NLP()
  df = pd.read_csv('some_file.csv')
  df['text'] = df['text'].apply(nlp.process)
  </code></pre>
  
  <p>Source: </p>
  
  <ol>
  <li><p><a href="https://github.com/miguelfzafra/Latest-News-Classifier" rel="noreferrer">https://github.com/miguelfzafra/Latest-News-Classifier</a></p></li>
  <li><p><a href="https://towardsdatascience.com/why-you-should-avoid-removing-stopwords-aa7a353d2a52" rel="noreferrer">https://towardsdatascience.com/why-you-should-avoid-removing-stopwords-aa7a353d2a52</a></p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T08:08:16.800Z
Answers.Text: Precisely can we remove "Free,
  Call,
  Text,
  Txt"
Answers.UserDisplayName: Pluviophile
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T08:13:00.333Z
Answers.Text: @Krishna Of course you can, as long as those words don't change the meaning of the text. Because if you train your model on data that its whole meaning has changed, then it is surely going to underperform. This happens very often, after removing stopwords the whole meaning of sentence changes.
Answers.UserDisplayName: Coderio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T08:21:22.767Z
Answers.Text: @Krishna Glad to help :)
Answers.UserDisplayName: Coderio
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-21T08:03:06.277Z
Answers.LastActivityDate: 2020-04-21T08:03:06.277Z
Answers.UserDisplayName: Coderio
Body: <p>Assume that I have a Dataframe with the text column.
  Problem: Classification / Prediction</p>
  <pre><code>    sms_text
  0   Go until jurong point, crazy.. Available only ...
  1   Ok lar... Joking wif u oni...
  2   Free entry in 2 a wkly comp to win FA Cup fina...
  3   U dun say so early hor... U c already then say...
  4   Nah I don't think he goes to usf, he lives aro...
  </code></pre>
  <p>After preprocessing the text
  <a href="https://i.sstatic.net/4EQWp.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4EQWp.png" alt="enter image description here" /></a></p>
  <p>From the above WordCloud, we can find the most frequent(occurred) words like</p>
  <pre><code>Free
  Call
  Text
  Txt
  </code></pre>
  <p>As these are the most frequent words and adds less importance in prediction/classification as they appear a lot. (My Opinion)</p>
  <blockquote>
  <p>My Question is
  Removing top frequent(most occurred) words will improve the model score?</p>
  <p>How does this impact on model performance?</p>
  <p>Is it ok to remove the most occurred words?</p>
  </blockquote>
  
Tags: <natural-language-processing><text-classification>
Title: Top Frequent occurrence word effect in Model Efficiency?
UserDisplayName: Pluviophile
ThreadText: # Top Frequent occurrence word effect in Model Efficiency?
  
  (Asked by: username_0 on 2020-04-16)
  
  Assume that I have a Dataframe with the text column.
  Problem: Classification / Prediction
  
  ```
      sms_text
  0   Go until jurong point, crazy.. Available only ...
  1   Ok lar... Joking wif u oni...
  2   Free entry in 2 a wkly comp to win FA Cup fina...
  3   U dun say so early hor... U c already then say...
  4   Nah I don't think he goes to usf, he lives aro...
  
  ```
  
  After preprocessing the text
  [![enter image description here](https://i.sstatic.net/4EQWp.png)](https://i.sstatic.net/4EQWp.png)
  
  From the above WordCloud, we can find the most frequent(occurred) words like
  
  ```
  Free
  Call
  Text
  Txt
  
  ```
  
  As these are the most frequent words and adds less importance in prediction/classification as they appear a lot. (My Opinion)
  
  > My Question is
  > Removing top frequent(most occurred) words will improve the model score?
  >
  > How does this impact on model performance?
  >
  > Is it ok to remove the most occurred words?
  
  **Tags:** natural-language-processing, text-classification
  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2020-04-21. Score: 4 | Accepted answer
  
  As far as I know, there are few **aspects** that would probably improve the model score:
  
  > 1. Normalization
  > 2. Lemmatization
  > 3. Stopwords removal (as you asked here)
  
  Based on your question, "is removing top frequent words (stopwords) will improve the model score?". The answer is, it depends on what kind of stopwords are you removing. The problem here is that if you do not remove stop words, the noise will increase in the dataset because of words like I, my, me, etc. Here is the comparison of those three **aspects** using **SVM Classifier**.
  
  [![Comparison using SVM Classifier](https://i.sstatic.net/BQavF.png)](https://i.sstatic.net/BQavF.png)
  
  You may see that without stopwords removal the Train Set Accuracy decreased to **94.81%** and the Test Set Accuracy decreased to **88.02%**. But, you should be careful about what kind of stopwords you are removing.
  
  If you are working with basic NLP techniques like BOW, Count Vectorizer or TF-IDF(Term Frequency and Inverse Document Frequency) then removing stopwords is a good idea because stopwords act like noise for these methods. If you working with LSTM’s or other models which capture the semantic meaning and the meaning of a word depends on the context of the previous text, then it becomes important not to remove stopwords.
  
  So, what's the solution?
  
  > You may want to create a Python package **nlppreprocess** which removes stops words that are not necessary. It also has some additional functionalities that can make cleaning of text fast. For example:
  
  ```
  from nlppreprocess import NLP
  import pandas as pd
  
  nlp = NLP()
  df = pd.read_csv('some_file.csv')
  df['text'] = df['text'].apply(nlp.process)
  
  ```
  
  Source:
  
  1. <https://github.com/miguelfzafra/Latest-News-Classifier>
  2. <https://towardsdatascience.com/why-you-should-avoid-removing-stopwords-aa7a353d2a52>
  
  > **Comments:**
  > **username_0**: Precisely can we remove "Free, Call, Text, Txt"  
  > **username_4**: @Krishna Of course you can, as long as those words don't change the meaning of the text. Because if you train your model on data that its whole meaning has changed, then it is surely going to underperform. This happens very often, after removing stopwords the whole meaning of sentence changes.  
  > **username_4**: @Krishna Glad to help :)  
  
  ---
  
  ### Answer by username_1 on 2020-04-21. Score: 4
  
  Based on my experience, I did 2 tasks that is proven to improve the accuracy/score of my model.
  
  1. Normalization
     * removing characters and symbols in a text
     * lowercase folding
  2. Stopwords removal (as what you asked)
  
  These process helped me improve my model since stopwords gave my model noise as I am using word frequency count to represent text.
  
  So based on what you asked, does stopwords removal improve score? It depends on your model. If you are using word count to represent text you may do stopwords removal to remove noise when doing text classification.
  
  
  ---
  
  ### Answer by username_2 on 2020-04-21. Score: 1
  
  Based on my project, there is how i clean and doing some preparation on the data.
  
  1. Delete specific charaters ('\r', '\n', '"',)
  2. Change into the lowercase
  3. Delete some symbols
  4. Lemmatization (change base word with wordnet)
  5. Delete stopwords.
  
  With these following step, i get some improvement accuracy score on my model.
  
  My project: <https://github.com/khaifagifari/NLP-Course-TelU>
  
  
  ---
  
  ### Answer by username_3 on 2020-04-16. Score: 1
  
  The technical term for these words is "[stop words](https://en.wikipedia.org/wiki/Stop_words)". Have a look at [Information Retrieval](https://en.wikipedia.org/wiki/Information_retrieval) and indexing (eg [TF/IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)) to make up your mind whether you want to remove them or not.
  
  
  ---
  
  