Answers.Body: <p>I think it just tries to emphasize that you need three, <em>non-intersecting</em>, chunks of datasets: training, validation, and test. So, you need some data in addition to the <em>training data</em> to tune hyperparameters. You can simply create the train/test/validation splits by sampling without replacement from an initial dataset. You don't need anything additional than this initial dataset.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-25T14:05:29.770Z
Answers.Text: The quote is from the introductory chapter, the word "additional" is used once in the OP's linked chapter, as is the word "dataset". Validation or dev  are not mentioned at all. I think it is less emphasis, than a simple introduction without using all the jargon.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-25T07:10:29.537Z
Answers.LastActivityDate: 2022-01-20T20:43:30.693Z
Answers.LastEditDate: 2022-01-20T20:43:30.693Z
Answers.UserDisplayName: SpiderRico
Answers.Body: <p>Surely you understand the concept of overfitting: if you are optimizing over hyperparameters under the same underlying data, then technically you are optimizing a new dataset <span class="math-container">$(\theta, D)$</span>, where <span class="math-container">$\theta$</span> are hyperparameters, and <span class="math-container">$D$</span> is the original dataset.</p>
  <p>This is not good, because in theory, <span class="math-container">$\theta$</span> should be independent of the underlying <em>specific</em> <span class="math-container">$D$</span> in question, so anything learned is likely to be overfitted to the specific <span class="math-container">$D$</span>. We mitigate this by requiring additional data for training hyperparameters, such that anything learned about <span class="math-container">$\theta$</span>  would be about <em>the underlying data-generating statistic</em> and not just any specfic <span class="math-container">$D$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-25T16:04:21.040Z
Answers.LastActivityDate: 2021-08-25T16:04:21.040Z
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Body: <p>Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following paragraph from <a href="https://www.deeplearningbook.org/contents/ml.html" rel="nofollow noreferrer">Chapter 5: Machine Learning Basics</a> from the book titled <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning</a> (by Aaron Courville et al.)</p>
  <blockquote>
  <p>Most machine learning algorithms have settings called hyperparameters,
  which must be determined outside the learning algorithm itself; we
  discuss how to set these using <strong>additional</strong> data.</p>
  </blockquote>
  <p>My doubt is about the usage of the word '<strong>additional</strong>' in the paragraph. Afaik, a small part of dataset under consideration is used to validate and hence in determining the hyperparameters, called as validation data. It is also a part of the dataset as training and testing data. You can check the <code>section 5.3</code> for more details.</p>
  <p>If yes, what is the need for the usage of the word 'additional'? Is it true that data for setting hyperparameters is taken outside of the underlying dataset?</p>
  
Tags: <datasets><hyper-parameters>
Title: Why data required for hyperparameter tuning is considered as an additional data?
UserDisplayName: hanugm
ThreadText: # Why data required for hyperparameter tuning is considered as an additional data?
  
  (Asked by: username_0 on 2021-08-25)
  
  Any parametric model may have parameters as well as hyperparameters. Learning algorithm deals with parameters and hyperparameters should be dealt outside learning algorithm. Consider the following paragraph from [Chapter 5: Machine Learning Basics](https://www.deeplearningbook.org/contents/ml.html) from the book titled [Deep Learning](https://www.deeplearningbook.org/) (by Aaron Courville et al.)
  
  > Most machine learning algorithms have settings called hyperparameters,
  > which must be determined outside the learning algorithm itself; we
  > discuss how to set these using **additional** data.
  
  My doubt is about the usage of the word '**additional**' in the paragraph. Afaik, a small part of dataset under consideration is used to validate and hence in determining the hyperparameters, called as validation data. It is also a part of the dataset as training and testing data. You can check the `section 5.3` for more details.
  
  If yes, what is the need for the usage of the word 'additional'? Is it true that data for setting hyperparameters is taken outside of the underlying dataset?
  
  **Tags:** datasets, hyper-parameters
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-25. Score: 1
  
  I think it just tries to emphasize that you need three, *non-intersecting*, chunks of datasets: training, validation, and test. So, you need some data in addition to the *training data* to tune hyperparameters. You can simply create the train/test/validation splits by sampling without replacement from an initial dataset. You don't need anything additional than this initial dataset.
  
  > **Comments:**
  > **username_2**: The quote is from the introductory chapter, the word "additional" is used once in the OP's linked chapter, as is the word "dataset". Validation or dev  are not mentioned at all. I think it is less emphasis, than a simple introduction without using all the jargon.  
  
  ---
  
  ### Answer by username_3 on 2021-08-25. Score: 0
  
  Surely you understand the concept of overfitting: if you are optimizing over hyperparameters under the same underlying data, then technically you are optimizing a new dataset $(\theta, D)$, where $\theta$ are hyperparameters, and $D$ is the original dataset.
  
  This is not good, because in theory, $\theta$ should be independent of the underlying *specific* $D$ in question, so anything learned is likely to be overfitted to the specific $D$. We mitigate this by requiring additional data for training hyperparameters, such that anything learned about $\theta$ would be about *the underlying data-generating statistic* and not just any specfic $D$.
  
  
  ---
  
  