Answers.Body: <p>In machine learning, you normally split your data into 3 parts (80-10-10%).</p>
  <p>The first part (80% of your initial data) is for the training of your ML model: this is known as the <strong>training dataset</strong>.</p>
  <p>The second part (10%) is the <strong>development set</strong> (or <em>dataset</em>), aka <strong>validation set</strong>. This is used as measuring your performance with various hyperparameters (e.g. in neural networks: layer size).</p>
  <p>After you found your best hyperparameters, you learn the model again on the training set, and then test it on your <strong>test dataset</strong> (10%), which the model has never seen before. Your result on the test data is now a good indicator how your model prediction quality is in the real world (because it was never optimized for this test data).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-13T16:34:23.543Z
Answers.LastActivityDate: 2021-01-21T00:11:37.703Z
Answers.LastEditDate: 2021-01-21T00:11:37.703Z
Answers.UserDisplayName: user3352632
Body: <p>This is a theoretical question. I am a newbie to artificial intelligence and machine learning, and the more I read the more I like this. So far, I have been reading about the evaluation of language models (I am focused on ASR), but I still don't get the concept of the <strong>development test sets</strong>.</p>
  <p>The clearest explanation I have come across is the following (taken from <a href="https://web.stanford.edu/%7Ejurafsky/slp3/3.pdf" rel="nofollow noreferrer">chapter 3</a> of the book <a href="https://web.stanford.edu/%7Ejurafsky/slp3/" rel="nofollow noreferrer">Speech and Language Processing (3rd ed. draft)</a> by Dan Jurafsky and James H. Martin)</p>
  <blockquote>
  <p>Sometimes we use a particular test set so often that we implicitly tune to its characteristics. We then need a fresh test set that is truly unseen. In such cases, we call the initial test set the development test set or, devset.</p>
  </blockquote>
  <p>In any case, I still don't understand why an additional test has to be used. In other words, why aren't training and test sets enough?</p>
  
Tags: <terminology><cross-validation><training-datasets><test-datasets><validation-datasets>
Title: What are "development test sets" used for?
UserDisplayName: little_mice
ThreadText: # What are "development test sets" used for?
  
  (Asked by: username_0 on 2018-03-13)
  
  This is a theoretical question. I am a newbie to artificial intelligence and machine learning, and the more I read the more I like this. So far, I have been reading about the evaluation of language models (I am focused on ASR), but I still don't get the concept of the **development test sets**.
  
  The clearest explanation I have come across is the following (taken from [chapter 3](https://web.stanford.edu/%7Ejurafsky/slp3/3.pdf) of the book [Speech and Language Processing (3rd ed. draft)](https://web.stanford.edu/%7Ejurafsky/slp3/) by Dan Jurafsky and James H. Martin)
  
  > Sometimes we use a particular test set so often that we implicitly tune to its characteristics. We then need a fresh test set that is truly unseen. In such cases, we call the initial test set the development test set or, devset.
  
  In any case, I still don't understand why an additional test has to be used. In other words, why aren't training and test sets enough?
  
  **Tags:** terminology, cross-validation, training-datasets, test-datasets, validation-datasets
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-03-13. Score: 6 | Accepted answer
  
  In machine learning, you normally split your data into 3 parts (80-10-10%).
  
  The first part (80% of your initial data) is for the training of your ML model: this is known as the **training dataset**.
  
  The second part (10%) is the **development set** (or *dataset*), aka **validation set**. This is used as measuring your performance with various hyperparameters (e.g. in neural networks: layer size).
  
  After you found your best hyperparameters, you learn the model again on the training set, and then test it on your **test dataset** (10%), which the model has never seen before. Your result on the test data is now a good indicator how your model prediction quality is in the real world (because it was never optimized for this test data).
  
  
  ---
  
  