Answers.Body: <p>Good question, it probably should, but I guess since it's the &quot;hello world&quot; of deep learning, they may have wanted to simplify it.</p>
  <p>But in your day to day use, you should split the training into train/validation, but keep test the way it is.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-25T15:49:08.643Z
Answers.LastActivityDate: 2023-10-25T15:49:08.643Z
Answers.UserDisplayName: Eoin Ó Coinnigh
Answers.Body: <p><strong>How is MNIST only providing the training and the test sets? What about the validation?</strong></p>
  <p>In order to perform cross-study comparison of model performance it makes a lot of sense to have a single test set for benchmarking.  This way, different investigators can compare their respective models in an &quot;apples to apples&quot; manner.  The test set is tested only once, as already mentioned, and is in a practical sense the final arbiter of model performance.  As an aside, the MNIST not only provides an important source of data for testing individual hypotheses, but it also serves to provide &quot;standards&quot;.  After all, NIST stands for National Institute of Standards and Technology.</p>
  <p><strong>The reason why a validation set is not parsed out (like the way the test set is).</strong></p>
  <p>Different investigators may want to perform different types of validations (hold out, n-fold cross-validation, leave-one-out cross-validation).  Thus, MNIST does not limit investigators from doing this by separating out a validation set.  MNIST leaves it up to the investigator to parse their &quot;training set&quot; into <em>yes</em>, a training set and validation set they way they prefer.  Unlike the test set, there is really no need to standardize the validation.  In contrast, test set does need to serve as a standard for benchmarking models.</p>
  <p><strong>NOTE:</strong>  To be clear, this does not mean that you should not perform a validation.  You absolutely must.  The only thing is that you must &quot;carve out&quot; the validation set in a manner of your choosing.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-23T22:49:34.940Z
Answers.LastActivityDate: 2022-10-23T22:49:34.940Z
Answers.UserDisplayName: Snehal Patel
Answers.Body: <p>There are a few reasons <code>why MNIST only provides training and test sets, and not a validation set</code>:</p>
  <ul>
  <li><p>Since, the MNIST dataset is intended to be a simple and straightforward benchmark for machine learning models, It is important to have a standard test set that can be used to compare different models.</p>
  </li>
  <li><p>Secondly, the MNIST dataset is well-known and well-studied.</p>
  </li>
  <li><p>Thirdly splitting it into three sets (<code>training</code>, <code>validation</code>, and <code>test</code>) would <strong>reduce</strong> the size of each set too much &amp; there is less need to have one.</p>
  </li>
  </ul>
  <p>The validation set is used to assess the performance of the model on unseen data. In this case, the validation set is not needed because the performance of the model can be assessed on the test set.</p>
  <p>Another common method is to use <a href="https://www.geeksforgeeks.org/cross-validation-machine-learning/" rel="nofollow noreferrer">cross-validation</a>, which is where the data is split into multiple sets and each set is used to train and test the model. This allows for a more accurate assessment of the model's performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T13:04:12.537Z
Answers.LastActivityDate: 2022-10-22T13:16:53.373Z
Answers.LastEditDate: 2022-10-22T13:16:53.373Z
Answers.UserDisplayName: Faizy
Answers.Body: <p>The <strong>test set</strong> should never be seen and ran once at the end of training.</p>
  <p>The <strong>validation set</strong> is used to help you select hyperparameters and it would be cheating to tune your model on the test set because you would be giving your model information about the test set. This would give your model an unfair advantage and skew the results; that simply means that if you are essentially using the test set for training data model, the model overfits to your test set, and will not generalize well to new, unseen data.</p>
  <p>For this reasons, the validation set <em>must</em> be a portion of the training data which is selected out and evaluated on during training so that you can do this. It's not necessary if you're not doing model selection.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T13:16:41.183Z
Answers.Text: Helpful answer. Can you edit it for explaining why "it would be cheating to tune your model to the test set"?
Answers.UserDisplayName: tail
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T13:18:14.863Z
Answers.Text: It would be cheating to tune your model to the test set because you would be giving your model information about the test set that it would not have during actual prediction. This would give your model an unfair advantage and skew the results.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T13:24:24.597Z
Answers.Text: that simply means that if you are essentially using the test set for training data model, the model overfits to your test set, and will not generalize well to new, unseen data.
Answers.UserDisplayName: Faizy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T19:31:00.093Z
Answers.Text: that's correct!
Answers.UserDisplayName: Winnie Xu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-22T12:49:19.197Z
Answers.LastActivityDate: 2022-10-24T11:50:50.940Z
Answers.LastEditDate: 2022-10-24T11:50:50.940Z
Answers.UserDisplayName: Winnie Xu
Body: <p>I was taught that, usually, a dataset has to be divided into three parts:</p>
  <ol>
  <li>Training set - for learning purposes</li>
  <li>Validation set - for picking the model which minimize the loss on this set</li>
  <li>Test test - for testing the performance of the model picked using metrics such as accuracy score</li>
  </ol>
  <p>How is <a href="http://yann.lecun.com/exdb/mnist/" rel="nofollow noreferrer">MNIST</a> only providing the training and the test sets? What about the validation?</p>
  
Tags: <datasets><training-datasets><mnist><validation-datasets>
Title: How is MNIST only providing the training and the test sets? What about the validation?
UserDisplayName: tail
ThreadText: # How is MNIST only providing the training and the test sets? What about the validation?
  
  (Asked by: username_0 on 2022-10-22)
  
  I was taught that, usually, a dataset has to be divided into three parts:
  
  1. Training set - for learning purposes
  2. Validation set - for picking the model which minimize the loss on this set
  3. Test test - for testing the performance of the model picked using metrics such as accuracy score
  
  How is [MNIST](http://yann.lecun.com/exdb/mnist/) only providing the training and the test sets? What about the validation?
  
  **Tags:** datasets, training-datasets, mnist, validation-datasets
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2022-10-22. Score: 5 | Accepted answer
  
  The **test set** should never be seen and ran once at the end of training.
  
  The **validation set** is used to help you select hyperparameters and it would be cheating to tune your model on the test set because you would be giving your model information about the test set. This would give your model an unfair advantage and skew the results; that simply means that if you are essentially using the test set for training data model, the model overfits to your test set, and will not generalize well to new, unseen data.
  
  For this reasons, the validation set *must* be a portion of the training data which is selected out and evaluated on during training so that you can do this. It's not necessary if you're not doing model selection.
  
  > **Comments:**
  > **username_0**: Helpful answer. Can you edit it for explaining why "it would be cheating to tune your model to the test set"?  
  > **username_3**: It would be cheating to tune your model to the test set because you would be giving your model information about the test set that it would not have during actual prediction. This would give your model an unfair advantage and skew the results.  
  > **username_3**: that simply means that if you are essentially using the test set for training data model, the model overfits to your test set, and will not generalize well to new, unseen data.  
  > **username_4**: that's correct!  
  
  ---
  
  ### Answer by username_1 on 2023-10-25. Score: 0
  
  Good question, it probably should, but I guess since it's the "hello world" of deep learning, they may have wanted to simplify it.
  
  But in your day to day use, you should split the training into train/validation, but keep test the way it is.
  
  
  ---
  
  ### Answer by username_2 on 2022-10-23. Score: 0
  
  **How is MNIST only providing the training and the test sets? What about the validation?**
  
  In order to perform cross-study comparison of model performance it makes a lot of sense to have a single test set for benchmarking. This way, different investigators can compare their respective models in an "apples to apples" manner. The test set is tested only once, as already mentioned, and is in a practical sense the final arbiter of model performance. As an aside, the MNIST not only provides an important source of data for testing individual hypotheses, but it also serves to provide "standards". After all, NIST stands for National Institute of Standards and Technology.
  
  **The reason why a validation set is not parsed out (like the way the test set is).**
  
  Different investigators may want to perform different types of validations (hold out, n-fold cross-validation, leave-one-out cross-validation). Thus, MNIST does not limit investigators from doing this by separating out a validation set. MNIST leaves it up to the investigator to parse their "training set" into *yes*, a training set and validation set they way they prefer. Unlike the test set, there is really no need to standardize the validation. In contrast, test set does need to serve as a standard for benchmarking models.
  
  **NOTE:** To be clear, this does not mean that you should not perform a validation. You absolutely must. The only thing is that you must "carve out" the validation set in a manner of your choosing.
  
  
  ---
  
  ### Answer by username_3 on 2022-10-22. Score: 0
  
  There are a few reasons `why MNIST only provides training and test sets, and not a validation set`:
  
  * Since, the MNIST dataset is intended to be a simple and straightforward benchmark for machine learning models, It is important to have a standard test set that can be used to compare different models.
  * Secondly, the MNIST dataset is well-known and well-studied.
  * Thirdly splitting it into three sets (`training`, `validation`, and `test`) would **reduce** the size of each set too much & there is less need to have one.
  
  The validation set is used to assess the performance of the model on unseen data. In this case, the validation set is not needed because the performance of the model can be assessed on the test set.
  
  Another common method is to use [cross-validation](https://www.geeksforgeeks.org/cross-validation-machine-learning/), which is where the data is split into multiple sets and each set is used to train and test the model. This allows for a more accurate assessment of the model's performance.
  
  
  ---
  
  