Answers.Body: <p><span class="math-container">$P(d)$</span> (aka evidence) is a probability of your data (observation) and is defined as follows:</p>
  <p><span class="math-container">$$
  P(d) = \sum_i P(d|c_i)P(c_i)
  $$</span></p>
  <p>for all classes <span class="math-container">$c$</span>.</p>
  <p>According to the book, <span class="math-container">$P(c)=\frac{N_c}{N_{doc}}$</span> and <span class="math-container">$P(d|c)$</span> is a likelihood and, applying the assumptions from the book, can be defined as <span class="math-container">$P(w_i
  |c)=\frac{\text{count}(w_i, c)+1}{\sum_{w \in V}\text{count}(w, c) + |V|}$</span>, where <span class="math-container">$V$</span> consists of the union of all the word types in all classes.</p>
  <p>Taking the example from Section 4.3 of the book,</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Dataset</th>
  <th>Cat</th>
  <th>Documents</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>Training</td>
  <td>-</td>
  <td>just plain boring</td>
  </tr>
  <tr>
  <td></td>
  <td>-</td>
  <td>entirely predictable and lacks energy</td>
  </tr>
  <tr>
  <td></td>
  <td>-</td>
  <td>no surprises and very few laughs</td>
  </tr>
  <tr>
  <td></td>
  <td>+</td>
  <td>very powerful</td>
  </tr>
  <tr>
  <td></td>
  <td>+</td>
  <td>the most fun film of the summer</td>
  </tr>
  <tr>
  <td>Test</td>
  <td>?</td>
  <td>predictable with no fun</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>we'll get:</p>
  <p><span class="math-container">$$
  P(-) = \frac{3}{5}\\
  P(+) = \frac{2}{5}\\
  P(S|−) = \frac{2\times 2\times 1}{34^3}\\
  P(S|+) = \frac{1\times 1 \times 2}{29^3}\\
  P(d) = P(S|−)P(−) + P(S|+)P(+) = 6.1\times 10^{−5} + 3.2\times 10^{−5}
  $$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T11:00:19.727Z
Answers.Text: Yes, this seems to make sense (as you're just applying marginalization), although I had not thought about it in this way, probably because I had not read that section of the book, so I didn't know whether the likelihood and evidence were given or how to compute them in this context. In any case, how would this be different from just computing the frequency of the document in the corpus (as the OP and I suggest in the comment above)?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T11:20:15.880Z
Answers.Text: @nbro The denominator is just a normalization constant and hence related to the likelihood times prior. Thus, the sum of the posterior for all categories is always equal to 1, which allows us to judge how confident we are in our predictions.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T11:26:57.190Z
Answers.Text: Yes, I know that the denominator is a normalization constant, but you're not answering my question. So, if someone asked you to estimate the probability of a document, how would you estimate it? Why wouldn't you just compute the frequency of the document in the corpus? That's my question. Yes, sure, you can compute P(d) by marginalizing over $c$, but why wouldn't you simply use the frequency? Anyway, how do you get from $(w_i \mid c)$ to the likelihood? It may be something you should explain more in detail.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T11:47:26.483Z
Answers.Text: @nbro Bayes' rule is difficult to comprehend. I would argue we cannot consider all documents with the same probability. This would only be true if we had exact duplicates in the dataset, which doesn't make any sense. As for $(w_i \mid c)$, it is taken from Eq. 4.12 in the book, which we compute as the fraction of times each word $w_i$ of a sentence $S$  appears among all words in all documents of topic $c$.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T12:33:17.120Z
Answers.Text: Bayes' rule is usually very easy to comprehend. Not sure what you mean by that. Of course, not all documents may have the same probability, but this is reflected in their potentially different frequency. Also not sure why you think that, generally, documents would have the same frequency. Finally, my question was: how do you relate $P(w_i \mid c)$ to $P(d \mid c)$, which is not explained in your answer.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-06T10:46:02.947Z
Answers.LastActivityDate: 2021-07-06T10:56:33.637Z
Answers.LastEditDate: 2021-07-06T10:56:33.637Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>In naive Bayes classification, we estimate the class of a document as follows</p>
  <p><span class="math-container">$$\hat{c} = \arg \max_{c \in C}  P(c \mid d) = \arg \max_{c \in C} \dfrac{ P(d \mid c)P(c) }{P(d)} $$</span></p>
  <p>It has been said in page 4 of <a href="https://web.stanford.edu/%7Ejurafsky/slp3/4.pdf" rel="nofollow noreferrer">this textbook</a> that we can ignore the probability of document since it remains constant across classes.</p>
  <blockquote>
  <p>We can conveniently simplify the above equation by dropping the denominator <span class="math-container">$p(d)$</span>.  This is possible because we will be computing <span class="math-container">$\dfrac{P(d \mid  c)P(c)}{P(d)}$</span>for each possible class. But <span class="math-container">$P(d)$</span> <strong>doesn't change for each class</strong>; we are always asking about the most likely class for the same document <span class="math-container">$d$</span>, which must have the same probability <span class="math-container">$P(d)$</span>.  Thus, we can choose the class that maximizes this simpler
  formula</p>
  <p><span class="math-container">$$\hat{c} = \arg \max_{c \in C}  P(c \mid d) = \arg \max_{c \in C}
   P(d \mid  c)P(c)  $$</span></p>
  </blockquote>
  <p>Since the value of the document does not influence the choice of the class, naive Bayes algorithm does not consider that.</p>
  <p>But, I want to know the value of <span class="math-container">$P(d)$</span>. Is it <span class="math-container">$\dfrac{1}{N}$</span>, if total number of documents are <span class="math-container">$N$</span>? How should I calculate <span class="math-container">$P(d)$</span>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-07-06T01:53:59.300Z
Comments.Text: I suppose that the frequency of the document in the corpus would be the natural estimate of the probability of that document appearing in the corpus. So, generally, you could have the probability $M_d/N$, where $M_d$ is the number of times the specific document appears in the corpus. So, yeah, I would say that you're right, but I am not an expert on this topic, so I don't know what people are doing in this context (and that's also why I didn't post this as a formal answer).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T11:25:04.980Z
Comments.Text: The probability of "the cat sat on the mat" is high; the probability of "fghkjrthwerwerwec" is low.
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <natural-language-processing><classification><probability><naive-bayes><probability-theory>
Title: How would the probability of a document $P(d)$ be computed in the Naive Bayes classifier?
UserDisplayName: hanugm
ThreadText: # How would the probability of a document $P(d)$ be computed in the Naive Bayes classifier?
  
  (Asked by: username_0 on 2021-07-04)
  
  In naive Bayes classification, we estimate the class of a document as follows
  
  $$\hat{c} = \arg \max\_{c \in C} P(c \mid d) = \arg \max\_{c \in C} \dfrac{ P(d \mid c)P(c) }{P(d)} $$
  
  It has been said in page 4 of [this textbook](https://web.stanford.edu/%7Ejurafsky/slp3/4.pdf) that we can ignore the probability of document since it remains constant across classes.
  
  > We can conveniently simplify the above equation by dropping the denominator $p(d)$. This is possible because we will be computing $\dfrac{P(d \mid c)P(c)}{P(d)}$for each possible class. But $P(d)$ **doesn't change for each class**; we are always asking about the most likely class for the same document $d$, which must have the same probability $P(d)$. Thus, we can choose the class that maximizes this simpler
  > formula
  >
  > $$\hat{c} = \arg \max\_{c \in C} P(c \mid d) = \arg \max\_{c \in C}
  > P(d \mid c)P(c) $$
  
  Since the value of the document does not influence the choice of the class, naive Bayes algorithm does not consider that.
  
  But, I want to know the value of $P(d)$. Is it $\dfrac{1}{N}$, if total number of documents are $N$? How should I calculate $P(d)$?
  
  **Tags:** natural-language-processing, classification, probability, naive-bayes, probability-theory
  > **Comments:**
  > **username_1**: I suppose that the frequency of the document in the corpus would be the natural estimate of the probability of that document appearing in the corpus. So, generally, you could have the probability $M_d/N$, where $M_d$ is the number of times the specific document appears in the corpus. So, yeah, I would say that you're right, but I am not an expert on this topic, so I don't know what people are doing in this context (and that's also why I didn't post this as a formal answer).  
  > **username_2**: The probability of "the cat sat on the mat" is high; the probability of "fghkjrthwerwerwec" is low.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-07-06. Score: 0
  
  $P(d)$ (aka evidence) is a probability of your data (observation) and is defined as follows:
  
  $$
  P(d) = \sum\_i P(d|c\_i)P(c\_i)
  $$
  
  for all classes $c$.
  
  According to the book, $P(c)=\frac{N\_c}{N\_{doc}}$ and $P(d|c)$ is a likelihood and, applying the assumptions from the book, can be defined as $P(w\_i
  |c)=\frac{\text{count}(w\_i, c)+1}{\sum\_{w \in V}\text{count}(w, c) + |V|}$, where $V$ consists of the union of all the word types in all classes.
  
  Taking the example from Section 4.3 of the book,
  
  | Dataset | Cat | Documents |
  | --- | --- | --- |
  | Training | - | just plain boring |
  |  | - | entirely predictable and lacks energy |
  |  | - | no surprises and very few laughs |
  |  | + | very powerful |
  |  | + | the most fun film of the summer |
  | Test | ? | predictable with no fun |
  
  we'll get:
  
  $$
  P(-) = \frac{3}{5}\\
  P(+) = \frac{2}{5}\\
  P(S|−) = \frac{2\times 2\times 1}{34^3}\\
  P(S|+) = \frac{1\times 1 \times 2}{29^3}\\
  P(d) = P(S|−)P(−) + P(S|+)P(+) = 6.1\times 10^{−5} + 3.2\times 10^{−5}
  $$
  
  > **Comments:**
  > **username_1**: Yes, this seems to make sense (as you're just applying marginalization), although I had not thought about it in this way, probably because I had not read that section of the book, so I didn't know whether the likelihood and evidence were given or how to compute them in this context. In any case, how would this be different from just computing the frequency of the document in the corpus (as the OP and I suggest in the comment above)?  
  > **username_3**: @nbro The denominator is just a normalization constant and hence related to the likelihood times prior. Thus, the sum of the posterior for all categories is always equal to 1, which allows us to judge how confident we are in our predictions.  
  > **username_1**: Yes, I know that the denominator is a normalization constant, but you're not answering my question. So, if someone asked you to estimate the probability of a document, how would you estimate it? Why wouldn't you just compute the frequency of the document in the corpus? That's my question. Yes, sure, you can compute P(d) by marginalizing over $c$, but why wouldn't you simply use the frequency? Anyway, how do you get from $(w_i \mid c)$ to the likelihood? It may be something you should explain more in detail.  
  > **username_3**: @nbro Bayes' rule is difficult to comprehend. I would argue we cannot consider all documents with the same probability. This would only be true if we had exact duplicates in the dataset, which doesn't make any sense. As for $(w_i \mid c)$, it is taken from Eq. 4.12 in the book, which we compute as the fraction of times each word $w_i$ of a sentence $S$  appears among all words in all documents of topic $c$.  
  > **username_1**: Bayes' rule is usually very easy to comprehend. Not sure what you mean by that. Of course, not all documents may have the same probability, but this is reflected in their potentially different frequency. Also not sure why you think that, generally, documents would have the same frequency. Finally, my question was: how do you relate $P(w_i \mid c)$ to $P(d \mid c)$, which is not explained in your answer.  
  
  ---
  
  