Answers.Body: <blockquote>
  <p>When training the model for NLP is it important to get rid of data which has &quot;bad semantic&quot; for learning process?</p>
  </blockquote>
  <p>No, this is backwards for classifiers such as the one you are training.*</p>
  <p>When training <em>any</em> model, it is usually important to expose it to the same kind of data that it will be operating on later.</p>
  <p>However, even more important than that, when training any model for a <em>purpose</em>, you first take some steps to turn the purpose into something you can assess. Typically a metric that measures the value of success or cost of errors that will occur in real-world scenario.</p>
  <p>So your starting point should be some test data that is representative of production data, and a way to assess how well your overall approach is working with that data. Ideally, this is <em>not</em> the same as your cross-validation data (which you may use 100s of times to tune individual models), but a separate set to determine whether one of your models is good enough to release.</p>
  <p>After that, you have some freedom to make complex pipelines or adjust the training data, and experiment with ideas for creating more sophisticated models. Remember with the final test, to test the entire pipeline end-to-end.</p>
  <p>In comments you suggest that the small refiner model will only be used on training dataset and not in production:</p>
  <blockquote>
  <p>In prod to be only the final model. &quot;Too general&quot; to be decided by the main model either by defining special category or I will expect inaccurate classification eg. 30% cat1, 25% cat2 and the rest distributed between the rest categories which will does not go over the defined threshold.</p>
  </blockquote>
  <p>Your problem here is that your refiner will have removed all the &quot;too general&quot; items from the training data. So the main model will not have examples to learn how to assign to that category, and you will have no control during training whether or not there is the inaccurate classification that you are expecting (the main model could equally well classify inaccurately with high confidence on data that is far outside its experience).</p>
  <p>However, if you follow the advice above and set a metric of hold-out production data, you <em>can</em> at least give your ideas a try with some confidence that you will understand whether or not they have worked.</p>
  <p>You also suggest:</p>
  <blockquote>
  <p>But indeed multiple models can be used in series as well.</p>
  </blockquote>
  <p>I recommend this pipeline approach, over trying to second-guess what the classifier will do with types of data that are significantly different from those it has been trained on. The end result may be two specialised models that work in series. Again you should set up your test data and test metric first. There could be little to no improvement over the simpler single model, in which case you may want to reject the pipeline approach as too complicated.</p>
  <hr />
  <p>* There may be <em>some</em> scenarios in which automated quality control refinement is useful. For instance training a generative model that you want to only output high quality text (for some definition of &quot;high quality&quot;). However, the situation presented in your question does not seem to fit this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T13:41:08.670Z
Answers.LastActivityDate: 2023-11-20T13:46:13.893Z
Answers.LastEditDate: 2023-11-20T13:46:13.893Z
Answers.UserDisplayName: Neil Slater
Body: <p>When training the model for NLP is it important to get rid of data which has &quot;bad semantic&quot; for learning process?</p>
  <p>My plan is to create a &quot;small model&quot; which can decide whether data used for training the final model are &quot;good&quot; or &quot;bad&quot;.</p>
  <p>For instance the small model will be made from &quot;good&quot; data example and &quot;bad&quot; data example.
  Later on all data which are going to be used to train a final big data model will be fist evaluated by the smaller one in order to increase overall quality of the data used in the final model.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-18T08:21:12.760Z
Comments.Text: What kind of NLP model are you building? I think the answer here will depend. If your finished product will be a sentiment analysis tool, probably a bad idea. If an autoregressive LLM, I don't know . . . But if you can say what it is, someone who knows the kind of model may be able to help
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-20T08:45:27.787Z
Comments.Text: @NeilSlater Building the automatic classification of the user help desk tickets (Queue, Category, Priority). Technology is OpenNLP from Apache. It works so far ok I still thinking how to make the model as good as possible. I believe the training data quality here is a key to gain a needed accuracy. Therefore filtering the "Too general tickets" might help. Indeed it will lead to the fact that the "general tickets" are going to be classified incorrectly but I think that is good. Because if the model do not provide accurate result it should be human handed anyhow rather then provide wrong result.
Comments.UserDisplayName: Milkmaid
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-20T09:04:56.377Z
Comments.Text: What will happen to tickets that are "too general" in the production system? Will you also be filtering them out from being categorised using the same tool there?
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-20T09:55:13.343Z
Comments.Text: In prod to be only the final model. "Too general" to be decided by the main model either by defining special category or I will expect inaccurate classification eg. 30% cat1, 25% cat2 and the rest distributed between the rest categories which will does not go over the defined threshold. But indeed multiple models can be used in series as well.
Comments.UserDisplayName: Milkmaid
Tags: <machine-learning><natural-language-processing><language-model>
Title: NLP "small" model to improve "big" model
UserDisplayName: Milkmaid
ThreadText: # NLP "small" model to improve "big" model
  
  (Asked by: username_0 on 2023-11-16)
  
  When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process?
  
  My plan is to create a "small model" which can decide whether data used for training the final model are "good" or "bad".
  
  For instance the small model will be made from "good" data example and "bad" data example.
  Later on all data which are going to be used to train a final big data model will be fist evaluated by the smaller one in order to increase overall quality of the data used in the final model.
  
  **Tags:** machine-learning, natural-language-processing, language-model
  > **Comments:**
  > **username_1**: What kind of NLP model are you building? I think the answer here will depend. If your finished product will be a sentiment analysis tool, probably a bad idea. If an autoregressive LLM, I don't know . . . But if you can say what it is, someone who knows the kind of model may be able to help  
  > **username_0**: @NeilSlater Building the automatic classification of the user help desk tickets (Queue, Category, Priority). Technology is OpenNLP from Apache. It works so far ok I still thinking how to make the model as good as possible. I believe the training data quality here is a key to gain a needed accuracy. Therefore filtering the "Too general tickets" might help. Indeed it will lead to the fact that the "general tickets" are going to be classified incorrectly but I think that is good. Because if the model do not provide accurate result it should be human handed anyhow rather then provide wrong result.  
  > **username_1**: What will happen to tickets that are "too general" in the production system? Will you also be filtering them out from being categorised using the same tool there?  
  > **username_0**: In prod to be only the final model. "Too general" to be decided by the main model either by defining special category or I will expect inaccurate classification eg. 30% cat1, 25% cat2 and the rest distributed between the rest categories which will does not go over the defined threshold. But indeed multiple models can be used in series as well.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-20. Score: 2 | Accepted answer
  
  > When training the model for NLP is it important to get rid of data which has "bad semantic" for learning process?
  
  No, this is backwards for classifiers such as the one you are training.\*
  
  When training *any* model, it is usually important to expose it to the same kind of data that it will be operating on later.
  
  However, even more important than that, when training any model for a *purpose*, you first take some steps to turn the purpose into something you can assess. Typically a metric that measures the value of success or cost of errors that will occur in real-world scenario.
  
  So your starting point should be some test data that is representative of production data, and a way to assess how well your overall approach is working with that data. Ideally, this is *not* the same as your cross-validation data (which you may use 100s of times to tune individual models), but a separate set to determine whether one of your models is good enough to release.
  
  After that, you have some freedom to make complex pipelines or adjust the training data, and experiment with ideas for creating more sophisticated models. Remember with the final test, to test the entire pipeline end-to-end.
  
  In comments you suggest that the small refiner model will only be used on training dataset and not in production:
  
  > In prod to be only the final model. "Too general" to be decided by the main model either by defining special category or I will expect inaccurate classification eg. 30% cat1, 25% cat2 and the rest distributed between the rest categories which will does not go over the defined threshold.
  
  Your problem here is that your refiner will have removed all the "too general" items from the training data. So the main model will not have examples to learn how to assign to that category, and you will have no control during training whether or not there is the inaccurate classification that you are expecting (the main model could equally well classify inaccurately with high confidence on data that is far outside its experience).
  
  However, if you follow the advice above and set a metric of hold-out production data, you *can* at least give your ideas a try with some confidence that you will understand whether or not they have worked.
  
  You also suggest:
  
  > But indeed multiple models can be used in series as well.
  
  I recommend this pipeline approach, over trying to second-guess what the classifier will do with types of data that are significantly different from those it has been trained on. The end result may be two specialised models that work in series. Again you should set up your test data and test metric first. There could be little to no improvement over the simpler single model, in which case you may want to reject the pipeline approach as too complicated.
  
  ---
  
  \* There may be *some* scenarios in which automated quality control refinement is useful. For instance training a generative model that you want to only output high quality text (for some definition of "high quality"). However, the situation presented in your question does not seem to fit this.
  
  
  ---
  
  