Answers.Body: <p>I refer you to <a href="https://www.cs.cmu.edu/~tom/" rel="nofollow noreferrer">Prof. Tom Mitchell</a>'s (the lecturer in the video) <a href="http://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf" rel="nofollow noreferrer">draft chapter</a> as the best explanation I could find. I will try to explain it in layman's terms here.</p>
  
  <p>Given a boolean problem, our logistic regression classifier will assign the label <span class="math-container">$Y=0$</span> iff <span class="math-container">$w_0 + \sum_{i=1}^{n}w_{i}X_{i} &gt; 0$</span> (from the video you posted; eqn. 18 in the draft chapter). Otherwise, it will assign the label <span class="math-container">$Y=1$</span>. Remember the goal of logistic regression is to learn the weights <span class="math-container">$w_0, w_1, ..., w_n$</span> that will give us a linear function that would seperate our data best.</p>
  
  <p>It can be shown that <span class="math-container">$w_0=\ln\frac{1-\pi}{\pi} + \sum_{i=1}^{n} \frac{\mu_{i1}^2 - \mu_{i0}^2}{2\sigma_{i}^2}$</span> and <span class="math-container">$w_i = \frac{\mu_{i0} - \mu_{i1}}{\sigma_{i}^2}$</span> (eqns. 19-23 in the draft chapter), where:</p>
  
  <ol>
  <li><span class="math-container">$\pi$</span> is <span class="math-container">$P(Y=1)$</span>,</li>
  <li><span class="math-container">$\mu_{i0}$</span> and <span class="math-container">$\mu_{i1}$</span> are the means of feature <span class="math-container">$X_i$</span> in classes <span class="math-container">$Y=0$</span> and <span class="math-container">$Y=1$</span>, respectively, and</li>
  <li><span class="math-container">$\sigma_i$</span> is the standard deviation of feature <span class="math-container">$X_i$</span></li>
  </ol>
  
  <p>This means that we can express our weights <span class="math-container">$w_0, w_1, ..., w_n$</span> in terms of the Gaussian Naive Bayes classifier's parameters. Now, here is the difference:</p>
  
  <p>Our goal is to find <span class="math-container">$P(Y=y_k|X_1 ... X_n)$</span>. Assuming conditional independence, <span class="math-container">$P(Y=y_k|X_1 ... X_n) \propto P(Y=y_k) \prod_{i}^n P(X_i | Y=y_k)$</span>.</p>
  
  <p><em>If we estimate <span class="math-container">$\pi$</span>, <span class="math-container">$\mu_{i0}$</span>, <span class="math-container">$\mu_{i1}$</span>, and <span class="math-container">$\sigma_i$</span>, then use those values to estimate <span class="math-container">$w_0, w_1, ..., w_n$</span>, then we use the conditional independence property directly in our calculations.</em> This is what the Naive Bayes classifier does.</p>
  
  <p><em>If we estimate <span class="math-container">$w_0, w_1, ..., w_n$</span> directly, by repeatedly minimizing the error of an assumed linear function, we are not strictly abiding by the conditional independence property, because we don't explicitly calculate an estimate of the conditional probability <span class="math-container">$P(X_i|Y=y_k)$</span>. We completely bypass that step.</em> This is what the Logistic Regression classifier does.</p>
  
  <p>From the section 4 of the draft chapter (GNB is short for Gaussian Naive Bayes):</p>
  
  <blockquote>
    <p>... Logistic Regression directly estimates the parameters of <span class="math-container">$P(Y|X)$</span>, whereas Naive Bayes directly estimates parameters for <span class="math-container">$P(Y)$</span> and <span class="math-container">$P(X|Y)$</span>... if the GNB assumptions hold, then asymptotically (as the number of training examples grows
    toward infinity) the GNB and Logistic Regression converge toward identical classifiers.</p>
  </blockquote>
  
  <p>Another way to understand this is that it has to do with the fact that <strong>Logistic Regression is a discriminative classifier</strong>, whereas <strong>Naive Bayes is a generative classifier</strong>. For more on that, read <a href="https://www.andrewng.org/" rel="nofollow noreferrer">Prof. Andrew Y. Ng</a> and <a href="https://people.eecs.berkeley.edu/~jordan/" rel="nofollow noreferrer">Prof. Michael I. Jordan</a>'s paper, <em><a href="https://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf" rel="nofollow noreferrer">On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes</a></em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-18T14:33:25.570Z
Answers.LastActivityDate: 2018-11-18T14:33:25.570Z
Answers.UserDisplayName: Abdul Rahman Dabbour
Body: <p>To my understanding, logistic regression is an extension of naive Bayes. </p>
  
  <p>Suppose <span class="math-container">$X = \{x_1, x_2, \dots, x_N \}$</span> and <span class="math-container">$Y = \{0, 1\}$</span>, each <span class="math-container">$x_i$</span> is i.i.d and <span class="math-container">$P(x_i \mid Y=y_k) \sim \mathcal{N}(\mu, \sigma^2)$</span> is a Gaussian distribution.</p>
  
  <p>In order to create a <em>linear decision surface</em>, we take the assumption of each pdf <span class="math-container">$P(x_i \mid y_k)$</span> having variance <span class="math-container">$\sigma^2$</span> independent of the value of <span class="math-container">$Y$</span>, i.e. 
  <span class="math-container">$\sigma_{i, k} = \sigma_i$</span> (<span class="math-container">$i \rightarrow x_i, k \rightarrow y_k$</span>).</p>
  
  <p>Finally, we end up learning the coefficients <span class="math-container">$(\omega_0, \omega_i)$</span>  that represent the <em>linear decision surface</em> in following equation (linear decision surface):</p>
  
  <p><span class="math-container">$$
  \frac{P(Y=0 \mid X)}{P(Y=1 \mid X)} = \omega_0 + \sum_{i} \omega \cdot x_i
  $$</span></p>
  
  <p>Even though the derivation of Linear Regression coefficients<span class="math-container">$(\omega_0, \omega_i)$</span> involves the assumption of conditional independent <span class="math-container">$x_i$</span> given <span class="math-container">$Y$</span>, <em>why is it said that learning these coefficients from training data are "somewhat more free" from conditional independence assumption, as compared to learning the regular Bayesian Distribution coefficients <span class="math-container">$\mu, \sigma$</span></em>?</p>
  
  <p>I came across this while following <a href="https://youtu.be/z_xPu9KrgCY?t=2008" rel="nofollow noreferrer">this course here</a>.</p>
  
Tags: <machine-learning><logistic-regression><naive-bayes>
Title: Is logistic regression more free from the conditional independence assumption than naive Bayes?
UserDisplayName: imflash217
ThreadText: # Is logistic regression more free from the conditional independence assumption than naive Bayes?
  
  (Asked by: username_0 on 2017-08-21)
  
  To my understanding, logistic regression is an extension of naive Bayes.
  
  Suppose $X = \{x\_1, x\_2, \dots, x\_N \}$ and $Y = \{0, 1\}$, each $x\_i$ is i.i.d and $P(x\_i \mid Y=y\_k) \sim \mathcal{N}(\mu, \sigma^2)$ is a Gaussian distribution.
  
  In order to create a *linear decision surface*, we take the assumption of each pdf $P(x\_i \mid y\_k)$ having variance $\sigma^2$ independent of the value of $Y$, i.e.
  $\sigma\_{i, k} = \sigma\_i$ ($i \rightarrow x\_i, k \rightarrow y\_k$).
  
  Finally, we end up learning the coefficients $(\omega\_0, \omega\_i)$ that represent the *linear decision surface* in following equation (linear decision surface):
  
  $$
  \frac{P(Y=0 \mid X)}{P(Y=1 \mid X)} = \omega\_0 + \sum\_{i} \omega \cdot x\_i
  $$
  
  Even though the derivation of Linear Regression coefficients$(\omega\_0, \omega\_i)$ involves the assumption of conditional independent $x\_i$ given $Y$, *why is it said that learning these coefficients from training data are "somewhat more free" from conditional independence assumption, as compared to learning the regular Bayesian Distribution coefficients $\mu, \sigma$*?
  
  I came across this while following [this course here](https://youtu.be/z_xPu9KrgCY?t=2008).
  
  **Tags:** machine-learning, logistic-regression, naive-bayes
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-11-18. Score: 1 | Accepted answer
  
  I refer you to [Prof. Tom Mitchell](https://www.cs.cmu.edu/~tom/)'s (the lecturer in the video) [draft chapter](http://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf) as the best explanation I could find. I will try to explain it in layman's terms here.
  
  Given a boolean problem, our logistic regression classifier will assign the label $Y=0$ iff $w\_0 + \sum\_{i=1}^{n}w\_{i}X\_{i} > 0$ (from the video you posted; eqn. 18 in the draft chapter). Otherwise, it will assign the label $Y=1$. Remember the goal of logistic regression is to learn the weights $w\_0, w\_1, ..., w\_n$ that will give us a linear function that would seperate our data best.
  
  It can be shown that $w\_0=\ln\frac{1-\pi}{\pi} + \sum\_{i=1}^{n} \frac{\mu\_{i1}^2 - \mu\_{i0}^2}{2\sigma\_{i}^2}$ and $w\_i = \frac{\mu\_{i0} - \mu\_{i1}}{\sigma\_{i}^2}$ (eqns. 19-23 in the draft chapter), where:
  
  1. $\pi$ is $P(Y=1)$,
  2. $\mu\_{i0}$ and $\mu\_{i1}$ are the means of feature $X\_i$ in classes $Y=0$ and $Y=1$, respectively, and
  3. $\sigma\_i$ is the standard deviation of feature $X\_i$
  
  This means that we can express our weights $w\_0, w\_1, ..., w\_n$ in terms of the Gaussian Naive Bayes classifier's parameters. Now, here is the difference:
  
  Our goal is to find $P(Y=y\_k|X\_1 ... X\_n)$. Assuming conditional independence, $P(Y=y\_k|X\_1 ... X\_n) \propto P(Y=y\_k) \prod\_{i}^n P(X\_i | Y=y\_k)$.
  
  *If we estimate $\pi$, $\mu\_{i0}$, $\mu\_{i1}$, and $\sigma\_i$, then use those values to estimate $w\_0, w\_1, ..., w\_n$, then we use the conditional independence property directly in our calculations.* This is what the Naive Bayes classifier does.
  
  *If we estimate $w\_0, w\_1, ..., w\_n$ directly, by repeatedly minimizing the error of an assumed linear function, we are not strictly abiding by the conditional independence property, because we don't explicitly calculate an estimate of the conditional probability $P(X\_i|Y=y\_k)$. We completely bypass that step.* This is what the Logistic Regression classifier does.
  
  From the section 4 of the draft chapter (GNB is short for Gaussian Naive Bayes):
  
  > ... Logistic Regression directly estimates the parameters of $P(Y|X)$, whereas Naive Bayes directly estimates parameters for $P(Y)$ and $P(X|Y)$... if the GNB assumptions hold, then asymptotically (as the number of training examples grows
  > toward infinity) the GNB and Logistic Regression converge toward identical classifiers.
  
  Another way to understand this is that it has to do with the fact that **Logistic Regression is a discriminative classifier**, whereas **Naive Bayes is a generative classifier**. For more on that, read [Prof. Andrew Y. Ng](https://www.andrewng.org/) and [Prof. Michael I. Jordan](https://people.eecs.berkeley.edu/~jordan/)'s paper, *[On Discriminative vs. Generative classifiers: A comparison of logistic regression and naive Bayes](https://ai.stanford.edu/~ang/papers/nips01-discriminativegenerative.pdf)*.
  
  
  ---
  
  