Answers.Body: <p>If using BART is already giving you good results, why do you need a new model?</p>
  <p>Not a rhetorical question. You might have good reasons for that. Training a model with less parameters optimized only on few classes is an example. But if you can't came up with good reasons then the answer is pretty straight forward: just use BART. Reason being that a model trained on another model annotations <strong>without any human in the loop</strong> will never perform better than the model used to automatically train the data. The first reason being error propagation, like in any other area of empirical science. And on a more technical level you can't really use tricks like using only predictions with a high confidence scores, cause deep learning models are basically never <a href="https://medium.com/analytics-vidhya/calibration-in-machine-learning-e7972ac93555" rel="nofollow noreferrer">calibrated</a>, hence you can't trust them and a human would still have to check them manually.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-05T17:23:04.743Z
Answers.Text: I see. The motivation for training a model from the BART predictions was so I could avoid concept drift, by constantly re-training the model based on new data BART predicts on. How do I deal with concept drift using BART, a pre-trained model?
Answers.UserDisplayName: Sanger Steel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-05T16:35:12.850Z
Answers.LastActivityDate: 2022-10-05T16:35:12.850Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm trying to set up a pipeline for my ML models to automatically re-train themselves whenever concept drift occurs to recalibrate to the new output distributions. However, I can't get ground-truth from my data without manual labeling, and I want an <em>automated</em> MLOps pipeline, so that no human-in-the-loop labeling is required.</p>
  <p>I was noticing that using zero-shot learning using BART was actually giving pretty impressive results with my label set on sample data I want to classify on.</p>
  <p>Suppose I were to use BART to manually label data, say, if it scored my label with a score of 0.95 or better, and then just used that labeled data to train my models.</p>
  <p>Is this &quot;cheating&quot;? Am I trying to cut a corner here that can't be cut? Will this create a meaningful model for my text data, or just cause my trained model to try to become a crude version of the BART model that provided the labels in the first place, and therefore just be worse than strictly using the BART model?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-05T16:38:29.777Z
Comments.Text: Seems possible: Google search for: automatic labeling artificial intelligence BART finds: https://github.com/CristianViorelPopa/BART-TL-topic-label-generation - leaving BART out of the search terms finds quite a few articles.
Comments.UserDisplayName: Rob
Tags: <text-classification><data-labelling><pretrained-models><zero-shot-learning><automated-machine-learning>
Title: Using a pre-trained model to generate labels to data to then train a model on
UserDisplayName: Sanger Steel
ThreadText: # Using a pre-trained model to generate labels to data to then train a model on
  
  (Asked by: username_0 on 2022-10-05)
  
  I'm trying to set up a pipeline for my ML models to automatically re-train themselves whenever concept drift occurs to recalibrate to the new output distributions. However, I can't get ground-truth from my data without manual labeling, and I want an *automated* MLOps pipeline, so that no human-in-the-loop labeling is required.
  
  I was noticing that using zero-shot learning using BART was actually giving pretty impressive results with my label set on sample data I want to classify on.
  
  Suppose I were to use BART to manually label data, say, if it scored my label with a score of 0.95 or better, and then just used that labeled data to train my models.
  
  Is this "cheating"? Am I trying to cut a corner here that can't be cut? Will this create a meaningful model for my text data, or just cause my trained model to try to become a crude version of the BART model that provided the labels in the first place, and therefore just be worse than strictly using the BART model?
  
  **Tags:** text-classification, data-labelling, pretrained-models, zero-shot-learning, automated-machine-learning
  
  > **Comments:**
  > **username_1**: Seems possible: Google search for: automatic labeling artificial intelligence BART finds: https://github.com/CristianViorelPopa/BART-TL-topic-label-generation - leaving BART out of the search terms finds quite a few articles.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-10-05. Score: 1
  
  If using BART is already giving you good results, why do you need a new model?
  
  Not a rhetorical question. You might have good reasons for that. Training a model with less parameters optimized only on few classes is an example. But if you can't came up with good reasons then the answer is pretty straight forward: just use BART. Reason being that a model trained on another model annotations **without any human in the loop** will never perform better than the model used to automatically train the data. The first reason being error propagation, like in any other area of empirical science. And on a more technical level you can't really use tricks like using only predictions with a high confidence scores, cause deep learning models are basically never [calibrated](https://medium.com/analytics-vidhya/calibration-in-machine-learning-e7972ac93555), hence you can't trust them and a human would still have to check them manually.
  
  > **Comments:**
  > **username_0**: I see. The motivation for training a model from the BART predictions was so I could avoid concept drift, by constantly re-training the model based on new data BART predicts on. How do I deal with concept drift using BART, a pre-trained model?  
  
  ---
  
  