Answers.Body: <p>In that particular competition, you can try using GAN to generate new data or adding noise to existing data. You can also use K-means algorithm. You can try using a smaller network and remove bias. May be you can use logistic regression to compare the result. You can also use a PCA method.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-04T06:30:31.570Z
Answers.Text: Upsampling (adding noise to labeled samples which results in new samples) can be an effective practice. However, I'm not sure about the rest of the above answer. 
  
  For example, I'm afraid 250 labeled samples may not be enough for a GAN as we have 200 feature columns. Or, according to the question, the best result is achieved by SVM (not an NN algorithm) so I'm not sure what "using a smaller network and remove bias" means in this context.
  
  It would be great if you can provide more details about the applications of K-Means and PCA here.
Answers.UserDisplayName: Borhan Kazimipour
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-04T06:36:54.170Z
Answers.Text: May be add some noise to the data to produce more
Answers.UserDisplayName: Clement
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-03T08:04:02.737Z
Answers.LastActivityDate: 2019-06-03T08:04:02.737Z
Answers.UserDisplayName: Clement
Body: <p>I'm trying to develop skills to deal with very small amounts of labeled samples (250 labeled/20000 total, 200 features) by practicing on Kaggle &quot;Don't Overfit&quot; <a href="https://www.kaggle.com/c/overfitting/data" rel="nofollow noreferrer">dataset</a> (Traget_Practice have provided all 20,000 Targets). I've read a ton of papers and articles on this topic, but everything I've tried did not improve simple regularized SVM results (best accuracy was 75 and AUC was 85) or any other algorithm result (LR, K-NN, NaiveBayes, RF, MLP). I believe the result can be better (on Leaderboard they go even over AUC 95)</p>
  <p>What I've tried without success:</p>
  <ul>
  <li><p><strong>Remove outliers</strong>  I've tried to remove 5%-10% outliers with EllipticEnvelope and with IsolationForest.</p>
  </li>
  <li><p><strong>Feature Selection</strong> I've tried RFE (with or without CV) + L1/L2 regularised LogisticRegression, and SelectKBest (with chi2).</p>
  </li>
  <li><p><strong>Semi-Supervised techniques</strong> I've tried co-training with different combinations of two complementary algorithms and :100-100: split features. I've also tried LabelSpreading, but I don't know how to provide the most uncertain samples (I tried predictions from other algorithms, but there were many mislabeled samples, hence was unsuccessful).</p>
  </li>
  <li><p><strong>Ensembling Classifiers</strong> StackingClassifier with all possible combinations of algorithms and this also didn't improve the result (the best is the same as SVM accuracy 75 and AUC 85).</p>
  </li>
  </ul>
  <p>Can anyone give me advice on what I'm doing wrong or what else to try?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-04T22:07:23.317Z
Comments.Text: Would it be easy to hand-label each data point? If so, you might try [active learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)).
Comments.UserDisplayName: Philip Raeisghasem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-06T11:08:10.057Z
Comments.Text: I know for that technique (Label Spreading work similar - only asking for most uncertain samples). But no, there is 200 continuous features which we don't know what they mean, so it is impossible to label them by hand.
Comments.UserDisplayName: FirePower
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-03T12:06:09.710Z
Comments.Text: In the forums on the competition they go into several of the approaches, i reccomend looking there
Comments.UserDisplayName: mshlis
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-25T12:17:42.733Z
Comments.Text: Just as an idea: Maybe you can try a neural network and initialize it with parameters returned from an autoencoder. This way the autoencoder can learn useful representations and you can start from there
Comments.UserDisplayName: amin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-25T15:56:32.020Z
Comments.Text: Take a look at seme-supervised learning (with GAN). For example, this [paper](https://arxiv.org/abs/1803.01541) or [that one](https://arxiv.org/abs/1703.02291). Also, here is my [results](https://github.com/karray/pathology_gan/blob/master/ssl/preliminary_report.ipynb). Let my know, if need more info on it, I'll post then an answer
Comments.UserDisplayName: Aray Karjauv
Tags: <machine-learning><semi-supervised-learning><data-labelling><labeled-datasets><unlabeled-datasets>
Title: How to deal with a small amount of labeled samples?
UserDisplayName: FirePower
ThreadText: # How to deal with a small amount of labeled samples?
  
  (Asked by: username_0 on 2019-05-04)
  
  I'm trying to develop skills to deal with very small amounts of labeled samples (250 labeled/20000 total, 200 features) by practicing on Kaggle "Don't Overfit" [dataset](https://www.kaggle.com/c/overfitting/data) (Traget\_Practice have provided all 20,000 Targets). I've read a ton of papers and articles on this topic, but everything I've tried did not improve simple regularized SVM results (best accuracy was 75 and AUC was 85) or any other algorithm result (LR, K-NN, NaiveBayes, RF, MLP). I believe the result can be better (on Leaderboard they go even over AUC 95)
  
  What I've tried without success:
  
  * **Remove outliers** I've tried to remove 5%-10% outliers with EllipticEnvelope and with IsolationForest.
  * **Feature Selection** I've tried RFE (with or without CV) + L1/L2 regularised LogisticRegression, and SelectKBest (with chi2).
  * **Semi-Supervised techniques** I've tried co-training with different combinations of two complementary algorithms and :100-100: split features. I've also tried LabelSpreading, but I don't know how to provide the most uncertain samples (I tried predictions from other algorithms, but there were many mislabeled samples, hence was unsuccessful).
  * **Ensembling Classifiers** StackingClassifier with all possible combinations of algorithms and this also didn't improve the result (the best is the same as SVM accuracy 75 and AUC 85).
  
  Can anyone give me advice on what I'm doing wrong or what else to try?
  
  **Tags:** machine-learning, semi-supervised-learning, data-labelling, labeled-datasets, unlabeled-datasets
  
  > **Comments:**
  > **username_1**: Would it be easy to hand-label each data point? If so, you might try [active learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)).  
  > **username_0**: I know for that technique (Label Spreading work similar - only asking for most uncertain samples). But no, there is 200 continuous features which we don't know what they mean, so it is impossible to label them by hand.  
  > **username_2**: In the forums on the competition they go into several of the approaches, i reccomend looking there  
  > **username_3**: Just as an idea: Maybe you can try a neural network and initialize it with parameters returned from an autoencoder. This way the autoencoder can learn useful representations and you can start from there  
  > **username_4**: Take a look at seme-supervised learning (with GAN). For example, this [paper](https://arxiv.org/abs/1803.01541) or [that one](https://arxiv.org/abs/1703.02291). Also, here is my [results](https://github.com/karray/pathology_gan/blob/master/ssl/preliminary_report.ipynb). Let my know, if need more info on it, I'll post then an answer  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_5 on 2019-06-03. Score: 1
  
  In that particular competition, you can try using GAN to generate new data or adding noise to existing data. You can also use K-means algorithm. You can try using a smaller network and remove bias. May be you can use logistic regression to compare the result. You can also use a PCA method.
  
  > **Comments:**
  > **username_6**: Upsampling (adding noise to labeled samples which results in new samples) can be an effective practice. However, I'm not sure about the rest of the above answer.   For example, I'm afraid 250 labeled samples may not be enough for a GAN as we have 200 feature columns. Or, according to the question, the best result is achieved by SVM (not an NN algorithm) so I'm not sure what "using a smaller network and remove bias" means in this context.  It would be great if you can provide more details about the applications of K-Means and PCA here.  
  > **username_5**: May be add some noise to the data to produce more  
  
  ---
  
  