Answers.Body: <p>Yes.. consider that neural networks are <em>function</em> approximators, and:</p>
  <blockquote>
  <p>In mathematics, a function from a set X to a set Y assigns to each element of X exactly one element of Y</p>
  </blockquote>
  <p>So, given the same X, you can only hope to have a specific Y, and if your dataset is ambiguous (such as in your case), then depending on the loss, you'll get some approximate answer (if derived from the MLE principle, probably the mean answer, but might change if you use MAP or regularization)</p>
  <p>So yes, no NN will ever be able to give a 0 loss to such problem (at least formalized as such)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-22T16:42:51.743Z
Answers.LastActivityDate: 2023-10-22T16:42:51.743Z
Answers.UserDisplayName: Alberto
Body: <p>Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence <code>aei</code>, with two occurrences preceding <code>o</code> and one preceding <code>u</code>, i.e., the dataset is:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: center;">Input</th>
  <th style="text-align: center;">Output</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: center;">aei</td>
  <td style="text-align: center;">o</td>
  </tr>
  <tr>
  <td style="text-align: center;">aei</td>
  <td style="text-align: center;">u</td>
  </tr>
  <tr>
  <td style="text-align: center;">aei</td>
  <td style="text-align: center;">o</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>In this case, the ideal probability distribution for the model's logits for <code>aei</code> would be <span class="math-container">$\sim 0.66$</span> for <code>o</code>, <span class="math-container">$\sim 0.33$</span> for <code>u</code>, and zero for other letters. In other words, when the model is input with <code>aei</code>, the ideal softmax of the logits would be <span class="math-container">$\sim 0.66$</span> for <code>o</code>, <span class="math-container">$\sim 0.33$</span> for <code>u</code>, and zero for other letters.</p>
  <p>Following this reasoning, the objective is to optimize the model's output for a given input to match the distribution of occurrences in the dataset.</p>
  <p>If this reasoning is correct, then we have the following ideal loss (cross-entropy):</p>
  <p><span class="math-container">$$
  L = \frac{- log\left(\frac{2}{3}\right) - log\left(\frac{1}{3}\right) - log\left(\frac{2}{3}\right)}{3} \approx 0.63
  $$</span></p>
  <p>Thus, <span class="math-container">$\sim 0.63$</span> is the smallest loss we can get with this dataset.</p>
  <p>Is my reasoning correct?</p>
  
Tags: <loss><language-model><cross-entropy><n-gram>
Title: What is the lowest possible loss for a language model?
UserDisplayName: ViniciusArruda
ThreadText: # What is the lowest possible loss for a language model?
  
  (Asked by: username_0 on 2023-10-19)
  
  Example: Suppose a character-level language model (three input letters to predict the next one), trained on a dataset which contains three instances of the sequence `aei`, with two occurrences preceding `o` and one preceding `u`, i.e., the dataset is:
  
  | Input | Output |
  | --- | --- |
  | aei | o |
  | aei | u |
  | aei | o |
  
  In this case, the ideal probability distribution for the model's logits for `aei` would be $\sim 0.66$ for `o`, $\sim 0.33$ for `u`, and zero for other letters. In other words, when the model is input with `aei`, the ideal softmax of the logits would be $\sim 0.66$ for `o`, $\sim 0.33$ for `u`, and zero for other letters.
  
  Following this reasoning, the objective is to optimize the model's output for a given input to match the distribution of occurrences in the dataset.
  
  If this reasoning is correct, then we have the following ideal loss (cross-entropy):
  
  $$
  L = \frac{- log\left(\frac{2}{3}\right) - log\left(\frac{1}{3}\right) - log\left(\frac{2}{3}\right)}{3} \approx 0.63
  $$
  
  Thus, $\sim 0.63$ is the smallest loss we can get with this dataset.
  
  Is my reasoning correct?
  
  **Tags:** loss, language-model, cross-entropy, n-gram
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-22. Score: 1
  
  Yes.. consider that neural networks are *function* approximators, and:
  
  > In mathematics, a function from a set X to a set Y assigns to each element of X exactly one element of Y
  
  So, given the same X, you can only hope to have a specific Y, and if your dataset is ambiguous (such as in your case), then depending on the loss, you'll get some approximate answer (if derived from the MLE principle, probably the mean answer, but might change if you use MAP or regularization)
  
  So yes, no NN will ever be able to give a 0 loss to such problem (at least formalized as such)
  
  
  ---
  
  