Answers.Body: <p>I'm not sure I understand the point.  The secret model will invariably have some loss.  This loss will be overlaid on the larger dataset as the secret model operates on it.  Whatever model is trained from this larger dataset will learn the labels as given including the loss.  If we then assume this final model also has a loss it likely learns a lossy version of the secret model's loss.  Which theoretically could be a more accurate version of the secret model but more than likely will be less.</p>
  <p>If I was forced to steel man this idea, given a known confusion one could use the secret model to label the larger dataset and focus specialists on correcting common confusion points.  This should be faster than labeling the whole dataset.  But without this step it's an invariably lossy model intimating an invariably lossy model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-09T19:57:30.400Z
Answers.Text: The question wasn't whether the procedure would work better than directly training on the benchmark test set but whether it is cheating.
Answers.UserDisplayName: Joachim Wagner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T15:37:20.443Z
Answers.LastActivityDate: 2024-01-29T15:37:20.443Z
Answers.UserDisplayName: foreverska
Answers.Body: <p>neural network are function approximators...</p>
  <p>What you are saying is that given a data distribution <span class="math-container">$D$</span>, you train a model <span class="math-container">$f$</span> on it, and then you you se <span class="math-container">$f$</span> to label a new dataset <span class="math-container">$D'$</span></p>
  <p>Now, you want to train a new model <span class="math-container">$g$</span> on <span class="math-container">$D'$</span>... however, <span class="math-container">$D'$</span> is labeled by <span class="math-container">$f$</span>, thus <span class="math-container">$g$</span> will approximate the function learned by <span class="math-container">$f$</span>, which is supposedly a good function for <span class="math-container">$D$</span>, thus <span class="math-container">$g$</span> will be close to <span class="math-container">$f$</span>, thus it's pretty much saying that <span class="math-container">$g$</span> will be trained on <span class="math-container">$D$</span></p>
  <hr />
  <p>I'm rephrasing with all the definitions:</p>
  <ul>
  <li><span class="math-container">$S$</span> is the specialist</li>
  <li><span class="math-container">$D$</span> the small dataset (benchmark)</li>
  <li><span class="math-container">$f$</span> is a neural network trained on <span class="math-container">$D$</span></li>
  <li><span class="math-container">$D'$</span> is a second dataset labeled by <span class="math-container">$f$</span></li>
  <li><span class="math-container">$g$</span> is a neural network trained on <span class="math-container">$D'$</span></li>
  </ul>
  <p>the labels of <span class="math-container">$D'$</span> are point-wise evaluations of <span class="math-container">$f$</span>, therefore when you will ask to <span class="math-container">$g$</span> to approximate them, you will have a new function (<span class="math-container">$g$</span>) that approximate <span class="math-container">$f$</span></p>
  <p>Now, depending on how related is <span class="math-container">$D'$</span> to <span class="math-container">$D$</span>, you will have more or less &quot;leakage&quot; from <span class="math-container">$f$</span> to <span class="math-container">$g$</span></p>
  <p>Conclusion: this method is effective only if <span class="math-container">$D'$</span> is similar to <span class="math-container">$D$</span>, but if that's the case, then you will have a lot of leakage from <span class="math-container">$f$</span> to <span class="math-container">$g$</span>, and thus you will have a final function <span class="math-container">$g$</span> that is almost trained on <span class="math-container">$D$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T11:35:41.693Z
Answers.Text: first, let's settle the terms:
  S - specialists that label the D, 
  D = benchmark dataset, 
  f= small secret model trained on benchmark dataset, 
  D' = real dataset labeled by the f, 
  g = real model trained on D 
  ----
  Did I get you right? (S is my addition for the upcoming question)
  If yes:
  The vanilla approach is that S  (specialists) labels both D and D' But in an ideal world the S and the f  would label D' similarly.
  So what is the difference? if you are right it should be prohibited for S (specialists) label both D and D')
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T12:57:53.600Z
Answers.Text: @Igor updated the answer
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-31T06:51:15.600Z
Answers.Text: Thank you. 
  But still, if you take an alternative scenario: the example where  the same S(specialist) labels both - D and D', and in both cases does it ideally, you will again get the same leakage, won't you?
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-31T13:03:43.803Z
Answers.Text: @Igor i mean, not really, but at that point, why do you need a small benchmark dataset, when you have a big one already labeled by an expert
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-31T14:58:01.967Z
Answers.Text: maybe there is a misunderstanding.
  There are 2 scenarios:
  
  1) The specialist labels only the benchmark, and I based on it train a "secret" model that is being used INSTEAD of the specialist to label the "real-dataset" to train a "- real model".   End of scenario 1.
  2 (standard pipeline)I use the specialist to label both: the benchmark (as in the previous scenario) and the real dataset (which in the previous scenario was labeled by a model substituting the specialist but a very good-ideal model ideally substituting him)
  Would you say that both cases are the same in terms of leakage amount?
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-09T20:09:22.073Z
Answers.Text: @Igor Scenario 2 is the standard procedure for creating a data set and splitting it into training and test data. Ideally, the data is annotated in random order. If the test data is annotated in a block, there will be some effect as annotators change behaviour over time and learn from performing the task. However, this effect is expected to be small. For difficult tasks, annotators usually are given a few "warm up" items to annotate that are discarded, or at least exempt from inclusion in the test set.
Answers.UserDisplayName: Joachim Wagner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-28T22:31:58.903Z
Answers.LastActivityDate: 2024-01-29T12:57:41.593Z
Answers.LastEditDate: 2024-01-29T12:57:41.593Z
Answers.UserDisplayName: Alberto
Body: <p>To assess our deep learning models (CNN) we have labeled a big benchmark dataset (it was labeled by specialists so it is kind of Ideal).
  I of course know that we <strong>do not want to train new models using the benchmark dataset</strong> since it would be &quot;cheating ourselves&quot;.
  The question is if we can train a small &quot;secret model&quot; based only on the benchmark, and use it only for faster labeling of other &quot;real&quot; datasets that unlike the benchmark dataset <strong>WILL</strong> be used to train our models.
  Does such an approach sound legitimate?</p>
  
Tags: <deep-learning><benchmarks><validation-datasets>
Title: Is it legitimate to train a model on a benchmark dataset and use this model only for labeling another datasets
UserDisplayName: Igor
ThreadText: # Is it legitimate to train a model on a benchmark dataset and use this model only for labeling another datasets
  
  (Asked by: username_0 on 2024-01-28)
  
  To assess our deep learning models (CNN) we have labeled a big benchmark dataset (it was labeled by specialists so it is kind of Ideal).
  I of course know that we **do not want to train new models using the benchmark dataset** since it would be "cheating ourselves".
  The question is if we can train a small "secret model" based only on the benchmark, and use it only for faster labeling of other "real" datasets that unlike the benchmark dataset **WILL** be used to train our models.
  Does such an approach sound legitimate?
  
  **Tags:** deep-learning, benchmarks, validation-datasets
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-01-29. Score: 1
  
  I'm not sure I understand the point. The secret model will invariably have some loss. This loss will be overlaid on the larger dataset as the secret model operates on it. Whatever model is trained from this larger dataset will learn the labels as given including the loss. If we then assume this final model also has a loss it likely learns a lossy version of the secret model's loss. Which theoretically could be a more accurate version of the secret model but more than likely will be less.
  
  If I was forced to steel man this idea, given a known confusion one could use the secret model to label the larger dataset and focus specialists on correcting common confusion points. This should be faster than labeling the whole dataset. But without this step it's an invariably lossy model intimating an invariably lossy model.
  
  > **Comments:**
  > **username_2**: The question wasn't whether the procedure would work better than directly training on the benchmark test set but whether it is cheating.  
  
  ---
  
  ### Answer by username_3 on 2024-01-28. Score: 1
  
  neural network are function approximators...
  
  What you are saying is that given a data distribution $D$, you train a model $f$ on it, and then you you se $f$ to label a new dataset $D'$
  
  Now, you want to train a new model $g$ on $D'$... however, $D'$ is labeled by $f$, thus $g$ will approximate the function learned by $f$, which is supposedly a good function for $D$, thus $g$ will be close to $f$, thus it's pretty much saying that $g$ will be trained on $D$
  
  ---
  
  I'm rephrasing with all the definitions:
  
  * $S$ is the specialist
  * $D$ the small dataset (benchmark)
  * $f$ is a neural network trained on $D$
  * $D'$ is a second dataset labeled by $f$
  * $g$ is a neural network trained on $D'$
  
  the labels of $D'$ are point-wise evaluations of $f$, therefore when you will ask to $g$ to approximate them, you will have a new function ($g$) that approximate $f$
  
  Now, depending on how related is $D'$ to $D$, you will have more or less "leakage" from $f$ to $g$
  
  Conclusion: this method is effective only if $D'$ is similar to $D$, but if that's the case, then you will have a lot of leakage from $f$ to $g$, and thus you will have a final function $g$ that is almost trained on $D$
  
  > **Comments:**
  > **username_0**: first, let's settle the terms: S - specialists that label the D,  D = benchmark dataset,  f= small secret model trained on benchmark dataset,  D' = real dataset labeled by the f,  g = real model trained on D  ---- Did I get you right? (S is my addition for the upcoming question) If yes: The vanilla approach is that S  (specialists) labels both D and D' But in an ideal world the S and the f  would label D' similarly. So what is the difference? if you are right it should be prohibited for S (specialists) label both D and D')  
  > **username_3**: @username_0 updated the answer  
  > **username_0**: Thank you.  But still, if you take an alternative scenario: the example where  the same S(specialist) labels both - D and D', and in both cases does it ideally, you will again get the same leakage, won't you?  
  > **username_3**: @username_0 i mean, not really, but at that point, why do you need a small benchmark dataset, when you have a big one already labeled by an expert  
  > **username_0**: maybe there is a misunderstanding. There are 2 scenarios:  1) The specialist labels only the benchmark, and I based on it train a "secret" model that is being used INSTEAD of the specialist to label the "real-dataset" to train a "- real model".   End of scenario 1. 2 (standard pipeline)I use the specialist to label both: the benchmark (as in the previous scenario) and the real dataset (which in the previous scenario was labeled by a model substituting the specialist but a very good-ideal model ideally substituting him) Would you say that both cases are the same in terms of leakage amount?  
  > **username_2**: @username_0 Scenario 2 is the standard procedure for creating a data set and splitting it into training and test data. Ideally, the data is annotated in random order. If the test data is annotated in a block, there will be some effect as annotators change behaviour over time and learn from performing the task. However, this effect is expected to be small. For difficult tasks, annotators usually are given a few "warm up" items to annotate that are discarded, or at least exempt from inclusion in the test set.  
  
  ---
  
  