Answers.Body: <p>I think that these terms may be used <em>inconsistently</em> across sources.</p>
  <p>If someone says <em>held-out dataset</em>, I would immediately think of a dataset that is not used for training, but can be used for anything else, validation (hyper-parameter tuning or early stopping) or testing; so, to determine what they are referring to, I would probably take into account the context.</p>
  <p>In your second quote, the <em>development set</em> seems to be used as a synonym for <em>validation dataset</em> (a more common name to refer to the same concept), i.e. the dataset used for early stopping or hyper-parameter optimization (see also <a href="https://ai.stackexchange.com/q/5658/2444">this</a>).</p>
  <p>So, my answer to your question in the title would be</p>
  <blockquote>
  <p>Yes, the heldout dataset can be used for validation or testing, but not because it's a special dataset, but because people may use this term to refer to either the validation dataset or the test dataset.</p>
  </blockquote>
  <p>Here's another example of the usage of the term <em>held-out set</em> to refer, in this case, to the validation dataset (section 1.6 of <a href="http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf" rel="nofollow noreferrer">this famous ML book</a>)</p>
  <blockquote>
  <p>The results above suggest a simple way of achieving this, namely by taking the available data and partitioning it into a training set, used to determine the coefficients <span class="math-container">$\mathbf{w}$</span>, and a <em>separate</em> validation set, also called a <strong>hold-out set</strong>, used to <em>optimize the model complexity</em></p>
  </blockquote>
  <p>Here's another example that shows that the term may be used inconsistently (emphasis mine, taken from <a href="https://www.deeplearningbook.org/contents/ml.html" rel="nofollow noreferrer">section 5.3</a> of the famous deep learning book by Goodfellow et al.). In fact, in that same section, they refer to the validation dataset, which is distinct from this <strong>held-out test set</strong> (so, in this case, the held-out set is used to refer only to the test set).</p>
  <blockquote>
  <p>Earlier we discussed how a <strong>held-out test set</strong>, composed of examples coming from the same distribution as the training set, can be used to estimate the generalization error of a learner, after the learning process has completed.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-03T03:06:24.400Z
Answers.LastActivityDate: 2021-07-03T03:06:24.400Z
Answers.UserDisplayName: nbro
Answers.Body: <p>In the sample dataset, the variable Sprint is the respondent's time (in seconds) to sprint a given distance, and Smoking is an indicator about whether or not the respondent smokes (0 = Nonsmoker, 1 = Past smoker, 2 = Current smoker). Use ANOVA to test if there is a statistically significant difference in sprint time with respect to smoking status. Sprint time will serve as the dependent variable, and smoking status will act as the independent variable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T00:43:51.610Z
Answers.LastActivityDate: 2023-05-04T00:43:51.610Z
Answers.UserDisplayName: sadiq umar
Body: <p>I came across a new term &quot;held-out corpora&quot; and I confused regarding its usage in the NLP domain</p>
  <p>Consider the following three paragraphs from <a href="https://web.stanford.edu/%7Ejurafsky/slp3/3.pdf" rel="nofollow noreferrer">N-gram Language Models</a></p>
  <p>#1: <strong>held-out corpora as a non-train data</strong></p>
  <blockquote>
  <p>For an intrinsic evaluation of a language model we need a test set. As
  with many of the statistical models in our field, the probabilities of
  an <span class="math-container">$n-$</span>gram model come from the corpus it is trained on, the training
  set or training corpus. We can then measure training set the quality
  of an n-gram model by its performance on some unseen data called the
  test set or test corpus.  <strong>We will also sometimes call test sets and
  other datasets that are not in our training sets held out corpora
  because we hold them out from the held out training data</strong>.</p>
  </blockquote>
  <p>This paragraph clearly says that held-out corpora can be used for either testing or validation or others except training.</p>
  <p>#2: <strong>development set or devset for hyperparameter tuning</strong></p>
  <blockquote>
  <p><strong>Sometimes we use a particular test set so often that we implicitly
  tune to its characteristics. We then need a fresh test set that is
  truly unseen. In such cases, we call the initial test set the
  development test set or,devset</strong>. How do we divide our data into
  training, development, and test sets? We want our test set to be as
  large as possible, since a small test set may be accidentally
  unrepresentative, but we also want as much training data as possible.
  At the minimum, we would want to pick the smallest test set that gives
  us enough statistical power to measure a statistically significant
  difference between two potential models. In practice, <strong>we often just
  divide our data into 80% training, 10% development, and 10% test.</strong>
  Given a large corpus that we want to divide into training and test,
  test data can either be taken from some continuous sequence of text
  inside the corpus, or we can remove smaller “stripes” of text from
  randomly selected parts of our corpus and combine them into a test
  set.</p>
  </blockquote>
  <p>This paragraph clearly says that development set is used for hyperparameter tuning.</p>
  <p>#3: <strong>held-out corpora for hyperparameter tuning</strong></p>
  <blockquote>
  <p>How are these <span class="math-container">$\lambda$</span> values set? Both the simple interpolation and
  conditional interpolation <span class="math-container">$\lambda'$</span>s are learned from a held-out
  corpus.  <strong>A held-out corpus is an additional training corpus that we
  use to set hyperparameters</strong> like these <span class="math-container">$\lambda$</span> values, by choosing
  the <span class="math-container">$\lambda$</span> values that maximize the likelihood of the held-out
  corpus.</p>
  </blockquote>
  <p>This paragraph is clearly saying that held-out corpus is used for hyper-parameter training.</p>
  <p><strong>I am interpreting or understanding the terms as follows</strong>:</p>
  <p><strong>Train corpus</strong> is used to train the model for learning parameters.</p>
  <p><strong>Test corpus</strong> is used for evaluating the model wrt parameters.</p>
  <p><strong>Development set</strong> is used for evaluating the model wrt hyperparameters.</p>
  <p><strong>Held-out corpus</strong> includes any corpus outside training corpus. So, it can be used for evaluating either parameters or hyperparameters.</p>
  <p>To be concise, informally, data = training data + held-out data = training data + development set + test data</p>
  <p>Is my understanding true? I got confusion because of paragraph 3, which says that held-out corpus is used (only) for learning the hyperparameters while paragraph 1 says that held-out corpus includes any corpus outside train corpus. Does held-out corpora include devset or same as devset?</p>
  
Tags: <natural-language-processing><terminology><books><test-datasets><validation-datasets>
Title: Are the held-out datasets used for testing, validation or both?
UserDisplayName: hanugm
ThreadText: # Are the held-out datasets used for testing, validation or both?
  
  (Asked by: username_0 on 2021-07-02)
  
  I came across a new term "held-out corpora" and I confused regarding its usage in the NLP domain
  
  Consider the following three paragraphs from [N-gram Language Models](https://web.stanford.edu/%7Ejurafsky/slp3/3.pdf)
  
  #1: **held-out corpora as a non-train data**
  
  > For an intrinsic evaluation of a language model we need a test set. As
  > with many of the statistical models in our field, the probabilities of
  > an $n-$gram model come from the corpus it is trained on, the training
  > set or training corpus. We can then measure training set the quality
  > of an n-gram model by its performance on some unseen data called the
  > test set or test corpus. **We will also sometimes call test sets and
  > other datasets that are not in our training sets held out corpora
  > because we hold them out from the held out training data**.
  
  This paragraph clearly says that held-out corpora can be used for either testing or validation or others except training.
  
  #2: **development set or devset for hyperparameter tuning**
  
  > **Sometimes we use a particular test set so often that we implicitly
  > tune to its characteristics. We then need a fresh test set that is
  > truly unseen. In such cases, we call the initial test set the
  > development test set or,devset**. How do we divide our data into
  > training, development, and test sets? We want our test set to be as
  > large as possible, since a small test set may be accidentally
  > unrepresentative, but we also want as much training data as possible.
  > At the minimum, we would want to pick the smallest test set that gives
  > us enough statistical power to measure a statistically significant
  > difference between two potential models. In practice, **we often just
  > divide our data into 80% training, 10% development, and 10% test.**
  > Given a large corpus that we want to divide into training and test,
  > test data can either be taken from some continuous sequence of text
  > inside the corpus, or we can remove smaller “stripes” of text from
  > randomly selected parts of our corpus and combine them into a test
  > set.
  
  This paragraph clearly says that development set is used for hyperparameter tuning.
  
  #3: **held-out corpora for hyperparameter tuning**
  
  > How are these $\lambda$ values set? Both the simple interpolation and
  > conditional interpolation $\lambda'$s are learned from a held-out
  > corpus. **A held-out corpus is an additional training corpus that we
  > use to set hyperparameters** like these $\lambda$ values, by choosing
  > the $\lambda$ values that maximize the likelihood of the held-out
  > corpus.
  
  This paragraph is clearly saying that held-out corpus is used for hyper-parameter training.
  
  **I am interpreting or understanding the terms as follows**:
  
  **Train corpus** is used to train the model for learning parameters.
  
  **Test corpus** is used for evaluating the model wrt parameters.
  
  **Development set** is used for evaluating the model wrt hyperparameters.
  
  **Held-out corpus** includes any corpus outside training corpus. So, it can be used for evaluating either parameters or hyperparameters.
  
  To be concise, informally, data = training data + held-out data = training data + development set + test data
  
  Is my understanding true? I got confusion because of paragraph 3, which says that held-out corpus is used (only) for learning the hyperparameters while paragraph 1 says that held-out corpus includes any corpus outside train corpus. Does held-out corpora include devset or same as devset?
  
  **Tags:** natural-language-processing, terminology, books, test-datasets, validation-datasets
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-07-03. Score: 1 | Accepted answer
  
  I think that these terms may be used *inconsistently* across sources.
  
  If someone says *held-out dataset*, I would immediately think of a dataset that is not used for training, but can be used for anything else, validation (hyper-parameter tuning or early stopping) or testing; so, to determine what they are referring to, I would probably take into account the context.
  
  In your second quote, the *development set* seems to be used as a synonym for *validation dataset* (a more common name to refer to the same concept), i.e. the dataset used for early stopping or hyper-parameter optimization (see also [this](https://ai.stackexchange.com/q/5658/2444)).
  
  So, my answer to your question in the title would be
  
  > Yes, the heldout dataset can be used for validation or testing, but not because it's a special dataset, but because people may use this term to refer to either the validation dataset or the test dataset.
  
  Here's another example of the usage of the term *held-out set* to refer, in this case, to the validation dataset (section 1.6 of [this famous ML book](http://users.isr.ist.utl.pt/%7Ewurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf))
  
  > The results above suggest a simple way of achieving this, namely by taking the available data and partitioning it into a training set, used to determine the coefficients $\mathbf{w}$, and a *separate* validation set, also called a **hold-out set**, used to *optimize the model complexity*
  
  Here's another example that shows that the term may be used inconsistently (emphasis mine, taken from [section 5.3](https://www.deeplearningbook.org/contents/ml.html) of the famous deep learning book by Goodfellow et al.). In fact, in that same section, they refer to the validation dataset, which is distinct from this **held-out test set** (so, in this case, the held-out set is used to refer only to the test set).
  
  > Earlier we discussed how a **held-out test set**, composed of examples coming from the same distribution as the training set, can be used to estimate the generalization error of a learner, after the learning process has completed.
  
  
  ---
  
  ### Answer by username_2 on 2023-05-04. Score: 0
  
  In the sample dataset, the variable Sprint is the respondent's time (in seconds) to sprint a given distance, and Smoking is an indicator about whether or not the respondent smokes (0 = Nonsmoker, 1 = Past smoker, 2 = Current smoker). Use ANOVA to test if there is a statistically significant difference in sprint time with respect to smoking status. Sprint time will serve as the dependent variable, and smoking status will act as the independent variable.
  
  
  ---
  
  