Answers.Body: <blockquote>
    <p>And that's all, we can infer P(x|y=c) and P(c) from the data. I don't see where the MLE shows its role.</p>
  </blockquote>
  
  <p>Maximum likelihood estimate is used for this very purpose, i.e. to estimate the conditional probability <span class="math-container">$p(x_j \mid y)$</span> and marginal probability <span class="math-container">$p(y)$</span> .</p>
  
  <hr>
  
  <p>In Naive Bayes Algorithm ,using the properties of <a href="https://en.wikipedia.org/wiki/Conditional_probability" rel="nofollow noreferrer">conditional probability</a>, we can estimate the joint probability</p>
  
  <p><span class="math-container">$$
  p(y, x_1, x_2, \dots, x_n) =  p(x_1, x_2, \dots, x_n \mid y) \; p(y)
  $$</span></p>
  
  <p>which is the same as<br>
  <span class="math-container">$$
  p(y=c|x)\propto p(x|y=c)p(y=c)
  $$</span></p>
  
  <p>What we need to estimate in here, are the conditional <span class="math-container">$p(x_j \mid y)$</span> and marginal <span class="math-container">$p(y)$</span> probabilities, and we use maximum likelihood for this. </p>
  
  <p>To be more precise: Assume that the <span class="math-container">$x$</span> is a feature vector <span class="math-container">$(x_1, x_2, ...x_n)$</span> - which means nothing more than the data point is a vector. In Naive Bayes Classification the assumption is that these features are independent of each other conditional on the class. So, in that case this term 
  <span class="math-container">$$p(x|y=c)$$</span> is replaced by <span class="math-container">$$\prod_{i=1}^{n} p(x_i|y=c)$$</span> -- this is because of the independence assumption that lets us simply multiply the probabilities. </p>
  
  <p>The problem of estimating the class to which the data point belongs reduces to the problem of maximizing the  <em>likelihood</em> <span class="math-container">$$\prod_{i=1}^{n} p(x_i|y=c)$$</span> -- which means assigning the data <span class="math-container">$(x_1, x_2, ...x_n)$</span>  to the class <span class="math-container">$k$</span> for which this likelihood is highest. This will give the same result as MLE which takes the form  of maximizing this quantity -- <span class="math-container">$$\prod_{i=1}^{n} p(x_i|\theta)$$</span> where <span class="math-container">$\theta$</span> are the assumed parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-09T06:32:02.193Z
Answers.LastActivityDate: 2019-12-03T12:20:51.110Z
Answers.LastEditDate: 2019-12-03T12:20:51.110Z
Answers.UserDisplayName: naive
Body: <p>I have found various references describing Naive Bayes and they all demonstrated that it used MLE for the calculation. However, this is my understanding:</p>
  
  <p><span class="math-container">$P(y=c|x)$</span> <span class="math-container">$\propto$</span> <span class="math-container">$P(x|y=c)P(y=c)$</span></p>
  
  <p>with <span class="math-container">$c$</span> is the class the model may classify <span class="math-container">$y$</span> as.</p>
  
  <p>And that's all, we can infer <span class="math-container">$P(x|y=c)$</span> and <span class="math-container">$P(c)$</span> from the data. I don't see where the MLE shows its role.</p>
  
Tags: <comparison><naive-bayes><maximum-likelihood>
Title: What is the relationship between MLE and naive Bayes?
UserDisplayName: Shrike Danny
ThreadText: # What is the relationship between MLE and naive Bayes?
  
  (Asked by: username_0 on 2019-05-09)
  
  I have found various references describing Naive Bayes and they all demonstrated that it used MLE for the calculation. However, this is my understanding:
  
  $P(y=c|x)$ $\propto$ $P(x|y=c)P(y=c)$
  
  with $c$ is the class the model may classify $y$ as.
  
  And that's all, we can infer $P(x|y=c)$ and $P(c)$ from the data. I don't see where the MLE shows its role.
  
  **Tags:** comparison, naive-bayes, maximum-likelihood
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-09. Score: 3
  
  > And that's all, we can infer P(x|y=c) and P(c) from the data. I don't see where the MLE shows its role.
  
  Maximum likelihood estimate is used for this very purpose, i.e. to estimate the conditional probability $p(x\_j \mid y)$ and marginal probability $p(y)$ .
  
  ---
  
  In Naive Bayes Algorithm ,using the properties of [conditional probability](https://en.wikipedia.org/wiki/Conditional_probability), we can estimate the joint probability
  
  $$
  p(y, x\_1, x\_2, \dots, x\_n) = p(x\_1, x\_2, \dots, x\_n \mid y) \; p(y)
  $$
  
  which is the same as  
  $$
  p(y=c|x)\propto p(x|y=c)p(y=c)
  $$
  
  What we need to estimate in here, are the conditional $p(x\_j \mid y)$ and marginal $p(y)$ probabilities, and we use maximum likelihood for this.
  
  To be more precise: Assume that the $x$ is a feature vector $(x\_1, x\_2, ...x\_n)$ - which means nothing more than the data point is a vector. In Naive Bayes Classification the assumption is that these features are independent of each other conditional on the class. So, in that case this term
  $$p(x|y=c)$$ is replaced by $$\prod\_{i=1}^{n} p(x\_i|y=c)$$ -- this is because of the independence assumption that lets us simply multiply the probabilities.
  
  The problem of estimating the class to which the data point belongs reduces to the problem of maximizing the *likelihood* $$\prod\_{i=1}^{n} p(x\_i|y=c)$$ -- which means assigning the data $(x\_1, x\_2, ...x\_n)$ to the class $k$ for which this likelihood is highest. This will give the same result as MLE which takes the form of maximizing this quantity -- $$\prod\_{i=1}^{n} p(x\_i|\theta)$$ where $\theta$ are the assumed parameters.
  
  
  ---
  
  