Answers.Body: <blockquote>
  <p>So where do I get the labels for my historical data?</p>
  </blockquote>
  <p>One option is to use the same procedure you used the first time, maybe manually?</p>
  <blockquote>
  <p>In that case, what's the point of training and serving a model then?</p>
  </blockquote>
  <p>You service the model to get predictions on new data, in an efficient manner (avoiding all the manual work!). As new data comes in, you could label some percentage of it manually and see how accurate your model actually is. If the accuracy isn't sufficient to your needs, you'll need to label more data and re-train the model.</p>
  <p>Or you can try different hyperparameters, train the model with only the initial batch of data, and re-evaluate its accuracy on this new batch of data. But in general you should do train-test-eval split anyway. Are you familiar with the k-fold validation procedure, and cross-validation?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T12:35:47.250Z
Answers.Text: I see, so basically the re-training should only happen when there is a need to improve the existing model (i.e. it doesn't happen on a regular basis) and the new training data is supposed to be manually labeled. That's the feedback i got back at stackoverflow, just wanted to re-check here as well. Thanks for reply!
Answers.UserDisplayName: mangusta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T12:39:12.837Z
Answers.Text: and yes, I read about k-fold validation, and as far as I understand, it could be applied on the initial batch, as well as any of subsequent batches
Answers.UserDisplayName: mangusta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T12:44:22.130Z
Answers.Text: I just heard so many times that the new data is labeled with the prediction from existing ML model, re-ingested back into ML input, and model is trained every day, using that new labeled data, but seems that it's a bad approach
Answers.UserDisplayName: mangusta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T12:49:23.537Z
Answers.Text: Personally I've used the procedure (on a computer vision task) that I manually label the initial data, train the model, do predictions on new data and save the label, but then CHECK whether the labels were correct or not! Maybe initially 60% is correct and you need to fix the rest, but as you repeat this procedure your "candidate" labels are more and more correct, and this is vastly more efficient than doing all the labeling by hand. In my case I was doing OCR on letters of varying sizes and orientations, so I could see the right answer in a glance, and move the image to the correct folder.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-18T12:08:55.123Z
Answers.LastActivityDate: 2024-09-18T12:08:55.123Z
Answers.UserDisplayName: NikoNyrh
Body: <p>I'm somewhat new to machine learning so there is one thing that I wanted to clarify.</p>
  <p>I was playing around with AWS Sagemaker, trained a model with some labeled data, deployed it to endpoint and set up Lambda for serving predictions.</p>
  <p>All good, but I want to re-train my model regularly, using, say, 1-week historical data.</p>
  <p>But my historical data is unlabeled which means it cannot be used for training. How do I label it?</p>
  <p>I initially thought that I can just use my model's predictions to label new (unlabeled) data but I have read that this not a good idea because it would simply assure my model about its accuracy, even though it might be far from being accurate.</p>
  <p>So where do I get the labels for my historical data?</p>
  <p>And if historical data cannot be labeled by model, then does it mean that it is supposed to be labeled manually? In that case, what's the point of training and serving a model then?</p>
  <p>As an example, let's take a fraudulent transaction detection. Ok, there is some initial data, labeled manually by someone who knows exactly if the transaction was fraudulent or not, and thus having 100% accuracy. Is it then supposed to be periodically and manually updated with additional 100%-accurate events?</p>
  
Tags: <machine-learning><training><classification><training-datasets>
Title: How to re-train ML model
UserDisplayName: mangusta
ThreadText: # How to re-train ML model
  
  (Asked by: username_0 on 2024-09-17)
  
  I'm somewhat new to machine learning so there is one thing that I wanted to clarify.
  
  I was playing around with AWS Sagemaker, trained a model with some labeled data, deployed it to endpoint and set up Lambda for serving predictions.
  
  All good, but I want to re-train my model regularly, using, say, 1-week historical data.
  
  But my historical data is unlabeled which means it cannot be used for training. How do I label it?
  
  I initially thought that I can just use my model's predictions to label new (unlabeled) data but I have read that this not a good idea because it would simply assure my model about its accuracy, even though it might be far from being accurate.
  
  So where do I get the labels for my historical data?
  
  And if historical data cannot be labeled by model, then does it mean that it is supposed to be labeled manually? In that case, what's the point of training and serving a model then?
  
  As an example, let's take a fraudulent transaction detection. Ok, there is some initial data, labeled manually by someone who knows exactly if the transaction was fraudulent or not, and thus having 100% accuracy. Is it then supposed to be periodically and manually updated with additional 100%-accurate events?
  
  **Tags:** machine-learning, training, classification, training-datasets
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-18. Score: 2 | Accepted answer
  
  > So where do I get the labels for my historical data?
  
  One option is to use the same procedure you used the first time, maybe manually?
  
  > In that case, what's the point of training and serving a model then?
  
  You service the model to get predictions on new data, in an efficient manner (avoiding all the manual work!). As new data comes in, you could label some percentage of it manually and see how accurate your model actually is. If the accuracy isn't sufficient to your needs, you'll need to label more data and re-train the model.
  
  Or you can try different hyperparameters, train the model with only the initial batch of data, and re-evaluate its accuracy on this new batch of data. But in general you should do train-test-eval split anyway. Are you familiar with the k-fold validation procedure, and cross-validation?
  
  > **Comments:**
  > **username_0**: I see, so basically the re-training should only happen when there is a need to improve the existing model (i.e. it doesn't happen on a regular basis) and the new training data is supposed to be manually labeled. That's the feedback i got back at stackoverflow, just wanted to re-check here as well. Thanks for reply!  
  > **username_0**: and yes, I read about k-fold validation, and as far as I understand, it could be applied on the initial batch, as well as any of subsequent batches  
  > **username_0**: I just heard so many times that the new data is labeled with the prediction from existing ML model, re-ingested back into ML input, and model is trained every day, using that new labeled data, but seems that it's a bad approach  
  > **username_1**: Personally I've used the procedure (on a computer vision task) that I manually label the initial data, train the model, do predictions on new data and save the label, but then CHECK whether the labels were correct or not! Maybe initially 60% is correct and you need to fix the rest, but as you repeat this procedure your "candidate" labels are more and more correct, and this is vastly more efficient than doing all the labeling by hand. In my case I was doing OCR on letters of varying sizes and orientations, so I could see the right answer in a glance, and move the image to the correct folder.  
  
  ---
  
  