Answers.Body: <p>Getting a high training accuracy on data without contradictions is nearly always possible, given a parametric function that can fit it in theory (enough flexibility and free parameters). It often does not require any special handling, such as the divide and conquer approach suggested in the question. Although that could possibly work.</p>
  <p>However, copying the training data accurately as possible is not the ultimate goal for most ML. Instead the more normal goal is to use the trained model to predict and make use of output values when they are not otherwise known - e.g. they predict future events, or things that take more effort to measure/calculate than the input data. For these kinds of use, the performance of ML on the training data is secondary to its performance on data outside of the training set.</p>
  <p>Forcing a perfect match to training data often leads to a phenomenon called overfitting, where performance on new unseen data is far worse than performance on the training data. So usually ML researchers focus their effort on improving results against validation and ultimately test data, which is data in the same format as training data including known target values. As the point is to not train against this data, it is not possible to force accuracy in the same way, and doing so would be counterproductive anyway: you want to know how good the ML is when used in reality, not just get a high score in a virtual space</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-29T12:08:49.563Z
Answers.LastActivityDate: 2023-10-29T12:08:49.563Z
Answers.UserDisplayName: Neil Slater
Body: <p>Suppose we have a classical machine learning problem. Say <span class="math-container">$m$</span> training examples and <span class="math-container">$n$</span> features with <span class="math-container">$m &gt;&gt; n$</span>. Suppose I find a great algorithm using automl or otherwise that gives 95% accuracy on the training set. What prevents us from learning from the 5% mistakes (if <span class="math-container">$0.05 m &gt;&gt;1$</span>) and training a new algorithm just on these 5% mistakes using another algorithm and so on till we get a small sample when this can't be done anymore?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-28T21:54:44.040Z
Comments.Text: To begin with, we don't really care about training accuracy, but rather, we want the model to generalize and perform well on unseen samples. Maximizing training accuracy is trivial anyway: just build a hashmap from samples to labels.
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-31T02:49:54.930Z
Comments.Text: Isn’t this the basis of boosting?
Comments.UserDisplayName: Dave
Tags: <machine-learning>
Title: Why can't one get extremely high accuracy with an machine learning algorithm?
UserDisplayName: wander95
ThreadText: # Why can't one get extremely high accuracy with an machine learning algorithm?
  
  (Asked by: username_0 on 2023-10-28)
  
  Suppose we have a classical machine learning problem. Say $m$ training examples and $n$ features with $m >> n$. Suppose I find a great algorithm using automl or otherwise that gives 95% accuracy on the training set. What prevents us from learning from the 5% mistakes (if $0.05 m >>1$) and training a new algorithm just on these 5% mistakes using another algorithm and so on till we get a small sample when this can't be done anymore?
  
  **Tags:** machine-learning
  > **Comments:**
  > **username_1**: To begin with, we don't really care about training accuracy, but rather, we want the model to generalize and perform well on unseen samples. Maximizing training accuracy is trivial anyway: just build a hashmap from samples to labels.  
  > **username_2**: Isn’t this the basis of boosting?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-10-29. Score: 3 | Accepted answer
  
  Getting a high training accuracy on data without contradictions is nearly always possible, given a parametric function that can fit it in theory (enough flexibility and free parameters). It often does not require any special handling, such as the divide and conquer approach suggested in the question. Although that could possibly work.
  
  However, copying the training data accurately as possible is not the ultimate goal for most ML. Instead the more normal goal is to use the trained model to predict and make use of output values when they are not otherwise known - e.g. they predict future events, or things that take more effort to measure/calculate than the input data. For these kinds of use, the performance of ML on the training data is secondary to its performance on data outside of the training set.
  
  Forcing a perfect match to training data often leads to a phenomenon called overfitting, where performance on new unseen data is far worse than performance on the training data. So usually ML researchers focus their effort on improving results against validation and ultimately test data, which is data in the same format as training data including known target values. As the point is to not train against this data, it is not possible to force accuracy in the same way, and doing so would be counterproductive anyway: you want to know how good the ML is when used in reality, not just get a high score in a virtual space
  
  
  ---
  
  