Answers.Body: <p>Because it's an ill-defined question.</p>
  <p>You might be 100% sure of the wrong answer :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-25T18:10:40.767Z
Answers.Text: You might, but there exist models where it's possible to specify how certain the model is, and that's valuable information even if there remain false answers that it's confident in.  Recall IBM Watson's Jeopardy answer of "What Is Toronto?????"
Answers.UserDisplayName: Ray
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-25T20:15:03.907Z
Answers.Text: @Ray no, you cannot. take a model that has a posterior distribution $p(x_T|x_{i<T})$ to be deterministic with 100% probability on the word "cat" no matter what. That prediction is 100% confident and 0% correct. For the models that give you some score, that usually is either the predictive entropy, the perplexity, or some score of the posterior distribution. However, changing the temperature of the softmax will induce these scores to change, even if the answer is the same, showing how they are definitely not reliable measures :)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-26T15:17:34.817Z
Answers.Text: "there exist models". There also exist plenty of models where it's not especially practical to give a meaningful confidence measure; I just wouldn't go so far as to call it an ill-defined question.
Answers.UserDisplayName: Ray
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-26T16:21:22.363Z
Answers.Text: @Ray then define what you mean by "how sure an LLM is" ;)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-26T19:20:00.667Z
Answers.Text: I wasn't really thinking LLMs specifically (hence my example of Watson). Language models aren't great for certainty measures, but more due to what the distribution is over, rather than any issues with temperature (we can always select the temperature to the one that minimizes some error metric over a validation set, after all, and while we *can* set the temperature arbitrarily high, by the time the resulting distribution starts looking anything like uniform, the model has long since stopped being useful).
Answers.UserDisplayName: Ray
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-22T10:42:25.520Z
Answers.LastActivityDate: 2024-09-22T10:42:25.520Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>For language models, the probabilities in question are (to first approximation) of the form &quot;Probability that word <em>i</em> is 'elephant' given the surrounding context&quot;, or more generally, the probability that if we randomly select a sentence/phrase/whatever of this length from the set of all possible sentences that we'll get this one. In this distribution, the sentences &quot;The answer is 'elephant'&quot;, &quot;An elephant&quot;, and &quot;one of those whatsises, like mammoths but less furry&quot; are all separate statements with separate probabilities. The model does not possess a single number that represents the probability that the <em>idea</em> is correct.</p>
  <p>Different types of model are more or less amenable to giving certainties of that form, and you could probably define some sort of model that provides a certainty measure over the set of (answer's semantic vector, question's semantic vector) pairs, but that's a decidedly nontrivial project in of itself, and doesn't naturally fall out of the existing math.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-25T18:07:41.340Z
Answers.LastActivityDate: 2024-09-25T18:14:45.650Z
Answers.LastEditDate: 2024-09-25T18:14:45.650Z
Answers.UserDisplayName: Ray
Body: <p>I am curious why the LLM model is not built to give a probabilistic answer. Whether people are working on it or is it not necessary to be accomplished.</p>
  
Tags: <natural-language-processing><bayesian-deep-learning>
Title: Why the company such as openai has not build probabilistic LLM which tells the user "How sure the model is about its answer"?
UserDisplayName: avenger
ThreadText: # Why the company such as openai has not build probabilistic LLM which tells the user "How sure the model is about its answer"?
  
  (Asked by: username_0 on 2024-09-22)
  
  I am curious why the LLM model is not built to give a probabilistic answer. Whether people are working on it or is it not necessary to be accomplished.
  
  **Tags:** natural-language-processing, bayesian-deep-learning
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-09-22. Score: 2
  
  Because it's an ill-defined question.
  
  You might be 100% sure of the wrong answer :)
  
  > **Comments:**
  > **username_2**: You might, but there exist models where it's possible to specify how certain the model is, and that's valuable information even if there remain false answers that it's confident in.  Recall IBM Watson's Jeopardy answer of "What Is Toronto?????"  
  > **username_1**: @username_2 no, you cannot. take a model that has a posterior distribution $p(x_T|x_{i<T})$ to be deterministic with 100% probability on the word "cat" no matter what. That prediction is 100% confident and 0% correct. For the models that give you some score, that usually is either the predictive entropy, the perplexity, or some score of the posterior distribution. However, changing the temperature of the softmax will induce these scores to change, even if the answer is the same, showing how they are definitely not reliable measures :)  
  > **username_2**: "there exist models". There also exist plenty of models where it's not especially practical to give a meaningful confidence measure; I just wouldn't go so far as to call it an ill-defined question.  
  > **username_1**: @username_2 then define what you mean by "how sure an LLM is" ;)  
  > **username_2**: I wasn't really thinking LLMs specifically (hence my example of Watson). Language models aren't great for certainty measures, but more due to what the distribution is over, rather than any issues with temperature (we can always select the temperature to the one that minimizes some error metric over a validation set, after all, and while we *can* set the temperature arbitrarily high, by the time the resulting distribution starts looking anything like uniform, the model has long since stopped being useful).  
  
  ---
  
  ### Answer by username_2 on 2024-09-25. Score: -1
  
  For language models, the probabilities in question are (to first approximation) of the form "Probability that word *i* is 'elephant' given the surrounding context", or more generally, the probability that if we randomly select a sentence/phrase/whatever of this length from the set of all possible sentences that we'll get this one. In this distribution, the sentences "The answer is 'elephant'", "An elephant", and "one of those whatsises, like mammoths but less furry" are all separate statements with separate probabilities. The model does not possess a single number that represents the probability that the *idea* is correct.
  
  Different types of model are more or less amenable to giving certainties of that form, and you could probably define some sort of model that provides a certainty measure over the set of (answer's semantic vector, question's semantic vector) pairs, but that's a decidedly nontrivial project in of itself, and doesn't naturally fall out of the existing math.
  
  
  ---
  
  