Answers.Body: <p><strong>Bernoulli naïve Bayes</strong></p>
  <p><span class="math-container">$P(x \mid c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}$</span></p>
  <p>Let's examine the example of document classification.<br />
  Let K different text classes and n different terms that our vocabulary contains. <span class="math-container">$x_i$</span>  are boolean variables (0, 1) expressing if the <span class="math-container">$i^{th}$</span> term exists in document <span class="math-container">$\mathbf{x}$</span>. <span class="math-container">$\mathbf{x}$</span> is a vector of dimension <span class="math-container">$n$</span>.</p>
  <p><span class="math-container">$P(x \mid c_k)$</span> is the probability that given the class <span class="math-container">$k$</span>, document <span class="math-container">$\mathbf{x}$</span> to be generated. The equation uses a common trick to represent a multivariate Bernoulli event model, taking into account that when <span class="math-container">$x_i = 1$</span>, then <span class="math-container">$1 - x_i = 0$</span> and inversely. In other words, for each term, it takes the probability that the document does contain this term or it does not.</p>
  <p><span class="math-container">$p_{ki}$</span> is the probability of class <span class="math-container">$c_k$</span> generating the term <span class="math-container">$x_i$</span>, that is it could be the prior probability a document that belongs to <span class="math-container">$k$</span> class contains this term of the vocabulary.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-11T22:59:47.150Z
Answers.LastActivityDate: 2021-12-13T09:13:42.347Z
Answers.LastEditDate: 2021-12-13T09:13:42.347Z
Answers.UserDisplayName: ddaedalus
Body: <p>I'm looking at the Bernoulli naïve Bayes classifier on Wikipedia and I understand Bayes theorem along with Gaussian naïve Bayes. However, when looking at how <span class="math-container">$P(x|c_k)$</span> is calculated, I don't understand it. The Wikipedia page says its calculated as follows</p>
  <p><span class="math-container">$$P(x|c_k) = \prod^{n}_{i=1} p^{x_i}_{ki} (1-p_{ki})^{(1-x_i)}. $$</span></p>
  <p>They mention that <span class="math-container">$p_{ki}$</span> is the probability of class <span class="math-container">$c_k$</span> generating the term <span class="math-container">$x_i$</span>, does that mean <span class="math-container">$P(x|c_k)$</span>? Because if so then that doesn't make sense since to calculate that we need to have calculated it already. So what is <span class="math-container">$p_{ki}$</span>?</p>
  <p>And in the first part, after the product symbol, are they raising this probability to the power pf <span class="math-container">$x_i$</span> or does that again just mean 'probability of class <span class="math-container">$c_k$</span> generating the term <span class="math-container">$x_i$</span>'?</p>
  <p>I also don't understand the intuition behind why or how this calculates <span class="math-container">$P(x|c_i)$</span>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T16:10:40.040Z
Comments.Text: Isn't $x$ a vector $[x_1, \dots, x_n]$?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T16:22:12.897Z
Comments.Text: @nbro Yes it's a feature vector although it can also be a matrix for larger data sets where the amount of rows is the 'number of samples', and the amount of columns is the 'number of features'.
Comments.UserDisplayName: Aguy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T16:23:14.110Z
Comments.Text: Therefore, that probability you're analysing can also be written as $P(x_1, \dots, x_n \mid c_k)$. Does this make things clearer? Now, what happens if you assume the observations are independent of each other?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T16:26:33.170Z
Comments.Text: @nbro I understand that. The problem I was facing is understanding what $p_{ki}$ is. And when in the formula they write $p_{ki}^{x_i}$ are they raising it to the power of $x_i$ or is it just saying that we're calculating $p_{ki}$ given $x_i$?
Comments.UserDisplayName: Aguy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T16:31:35.750Z
Comments.Text: It's been a while since I had to deal with naive Bayes. so I cannot help you more unless I read about it again (maybe I'll do that later), but have a look at the binomial distribution. The formulas look very similar. Anyway, I am pretty sure that this is an exponentiation operation, otherwise, what would the exponent $1 - x_i$ mean? In any case, $x_i$ must be encoded as a number. Maybe they mean $1 - i$ (rather than $1 - x_i$). Similarly, maybe they meant $p_{ki}^i$.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-05-03T16:33:42.577Z
Comments.Text: @nbro That exponent thing makes sense, just wanted to check that as well. And thanks for the help you're able to give at the moment. And I like I said I understand the rest and how to implement it except for what $p_{ki}$ is and how to calculate it.
Comments.UserDisplayName: Aguy
Tags: <naive-bayes><probability-theory><bayes-theorem><bayesian-probability>
Title: Understanding how to calculate $P(x|c_k)$ for the Bernoulli naïve Bayes classifier
UserDisplayName: Aguy
ThreadText: # Understanding how to calculate $P(x|c_k)$ for the Bernoulli naïve Bayes classifier
  
  (Asked by: username_0 on 2020-05-03)
  
  I'm looking at the Bernoulli naïve Bayes classifier on Wikipedia and I understand Bayes theorem along with Gaussian naïve Bayes. However, when looking at how $P(x|c\_k)$ is calculated, I don't understand it. The Wikipedia page says its calculated as follows
  
  $$P(x|c\_k) = \prod^{n}\_{i=1} p^{x\_i}\_{ki} (1-p\_{ki})^{(1-x\_i)}. $$
  
  They mention that $p\_{ki}$ is the probability of class $c\_k$ generating the term $x\_i$, does that mean $P(x|c\_k)$? Because if so then that doesn't make sense since to calculate that we need to have calculated it already. So what is $p\_{ki}$?
  
  And in the first part, after the product symbol, are they raising this probability to the power pf $x\_i$ or does that again just mean 'probability of class $c\_k$ generating the term $x\_i$'?
  
  I also don't understand the intuition behind why or how this calculates $P(x|c\_i)$.
  
  **Tags:** naive-bayes, probability-theory, bayes-theorem, bayesian-probability
  
  > **Comments:**
  > **username_1**: Isn't $x$ a vector $[x_1, \dots, x_n]$?  
  > **username_0**: @username_1 Yes it's a feature vector although it can also be a matrix for larger data sets where the amount of rows is the 'number of samples', and the amount of columns is the 'number of features'.  
  > **username_1**: Therefore, that probability you're analysing can also be written as $P(x_1, \dots, x_n \mid c_k)$. Does this make things clearer? Now, what happens if you assume the observations are independent of each other?  
  > **username_0**: @username_1 I understand that. The problem I was facing is understanding what $p_{ki}$ is. And when in the formula they write $p_{ki}^{x_i}$ are they raising it to the power of $x_i$ or is it just saying that we're calculating $p_{ki}$ given $x_i$?  
  > **username_1**: It's been a while since I had to deal with naive Bayes. so I cannot help you more unless I read about it again (maybe I'll do that later), but have a look at the binomial distribution. The formulas look very similar. Anyway, I am pretty sure that this is an exponentiation operation, otherwise, what would the exponent $1 - x_i$ mean? In any case, $x_i$ must be encoded as a number. Maybe they mean $1 - i$ (rather than $1 - x_i$). Similarly, maybe they meant $p_{ki}^i$.  
  > **username_0**: @username_1 That exponent thing makes sense, just wanted to check that as well. And thanks for the help you're able to give at the moment. And I like I said I understand the rest and how to implement it except for what $p_{ki}$ is and how to calculate it.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-05-11. Score: 1 | Accepted answer
  
  **Bernoulli naïve Bayes**
  
  $P(x \mid c\_k) = \prod^{n}\_{i=1} p^{x\_i}\_{ki} (1-p\_{ki})^{(1-x\_i)}$
  
  Let's examine the example of document classification.  
  Let K different text classes and n different terms that our vocabulary contains. $x\_i$ are boolean variables (0, 1) expressing if the $i^{th}$ term exists in document $\mathbf{x}$. $\mathbf{x}$ is a vector of dimension $n$.
  
  $P(x \mid c\_k)$ is the probability that given the class $k$, document $\mathbf{x}$ to be generated. The equation uses a common trick to represent a multivariate Bernoulli event model, taking into account that when $x\_i = 1$, then $1 - x\_i = 0$ and inversely. In other words, for each term, it takes the probability that the document does contain this term or it does not.
  
  $p\_{ki}$ is the probability of class $c\_k$ generating the term $x\_i$, that is it could be the prior probability a document that belongs to $k$ class contains this term of the vocabulary.
  
  
  ---
  
  