Answers.Body: <p>The relationship between the axes of graph (1) and your variables <span class="math-container">$x$</span> and <span class="math-container">$y$</span> is not clear, so this generalized answer may be helpful or useless.</p>
  
  <p>From graph (1) it appears that the correlation coefficient <span class="math-container">$\mathcal{C}$</span> of a quadratic fit of data set <span class="math-container">$\mathcal{S}$</span> would be much better.  Consider <span class="math-container">$y_1$</span> and <span class="math-container">$y_2$</span> approximations of <span class="math-container">$y$</span>.</p>
  
  <p><span class="math-container">$$
  \mathcal{C} (y_2, a, b, c, \mathcal{S}) &gt; \mathcal{C} (y_1, a, b, \mathcal{S}) \\
  y_2 = ax^2 + bx + c \\
  y_1 = ax + b
  $$</span></p>
  
  <p>To achieve a more nearly uniform distribution, perform a least squares fit for <span class="math-container">$y_2$</span> against <span class="math-container">$y$</span> on <span class="math-container">$\mathcal{S}$</span> to obtain <span class="math-container">$(a, b, c)$</span>.  Then find a mapping function that produces <span class="math-container">$y'$</span> and use it where the uniform distribution is desired.  A reasonable approximation is simply this.</p>
  
  <p><span class="math-container">$$y' = \frac{y}{y_2(x)}$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-20T13:35:39.720Z
Answers.LastActivityDate: 2019-08-20T13:35:39.720Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>While analyzing the data for a given problem set, I came across a few distributions which are not Gaussian in nature. They are not even uniform or Gamma distributions(so that I can write a function, plug the parameters and calculate the "Likelihood probability" and solve it using Bayes classification method). I got a set of a few absurd looking PDFs and I am wondering how should I define them mathematically so that I can plug the parameters and calculate the likelihood probability. </p>
  
  <p>The set of PDFs/Distributions that I got are the following and I am including some solutions that I intend to use. Please comment on their validity:</p>
  
  <p>1)<a href="https://i.sstatic.net/P5GYb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/P5GYb.png" alt="enter image description here"></a></p>
  
  <p>The distribution looks like:</p>
  
  <p><span class="math-container">$ y = ax +b $</span> from <span class="math-container">$ 0.8&lt;x&lt;1.5 $</span></p>
  
  <p>How to programmatically calculate</p>
  
  <pre><code>1. The value of x where the pdf starts
  2. The value of x where the pdf ends
  3. The value of y where the pdf starts
  4. The value of y where the pdf ends
  </code></pre>
  
  <p>However, I would have liked it better to have a generic distribution for this form of graphs so that I can plug the parameters to calculate the probability.</p>
  
  <p>2)<a href="https://i.sstatic.net/4X0K0.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4X0K0.png" alt="enter image description here"></a></p>
  
  <p>This PDF looks neither uniform nor Gaussian. What kind of distribution should I consider it roughly?</p>
  
  <p>3)<a href="https://i.sstatic.net/8pqnM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8pqnM.png" alt="enter image description here"></a></p>
  
  <p>I can divide this graph into three segments. The first segment is from <span class="math-container">$2&lt;x&lt;3$</span> with a steep slope, the second segment is from <span class="math-container">$3&lt;x&lt;6$</span> with a moderate sope and the third segment is from <span class="math-container">$6&lt;x&lt;8$</span> with a high negative slope.</p>
  
  <p>How to programmatically calculate</p>
  
  <pre><code> 1. the values of x where the graph changes its slope.
   2. the values of y where the graph changes its slope.
  </code></pre>
  
  <p>4)<a href="https://i.sstatic.net/pAQUv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/pAQUv.png" alt="enter image description here"></a></p>
  
  <p>This looks like two Gaussian densities with different mean superimposed together. But then the question arises, how do we find these two individual Gaussian densities?</p>
  
  <p>The following code may help:</p>
  
  <pre><code>variable1=nasa1['PerihelionArg'][nasa1.PerihelionArg&gt;190] 
  variable2=nasa1['PerihelionArg'][nasa1.PerihelionArg&lt;190] 
  </code></pre>
  
  <p>Find mean and variance of <code>variable1</code> and <code>variable2</code>, find the corresponding PDFs. Define the overall PDF with a suitable range of <span class="math-container">$x$</span>.</p>
  
  <p>5)<a href="https://i.sstatic.net/SyrFO.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/SyrFO.png" alt="enter image description here"></a></p>
  
  <p>This can be estimated as a Gamma distribution. We can find the mean and variance, calculate <span class="math-container">$\alpha$</span> and <span class="math-container">$\beta$</span> and finally calculate the PDF.</p>
  
  <p>It would be very helpful if someone could give their insights on the above analysis, its validity, and correctness and their suggestions regarding how problems such as these should be dealt with.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-06T06:06:02.410Z
Comments.Text: I am trying to train the classifier using Bayes theorem. Suppose for a given set of input, I want to determine if that asteroid is hazardous or not. For that, we need to calculate the probability P(Perihelion Time/Asteroid is Hazardous), that is, what is the probability that the asteroid takes the particular "Perihelion time"  (mentioned in the test input) given that the asteroid is "hazardous".
Comments.UserDisplayName: Soumee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-06T06:06:22.050Z
Comments.Text: So, to calculate P(Perihelion Time/Asteroid is Hazardous) we can segregate those values of Perihelion Time for which the asteroid is hazardous, compute its mean and variance and draw the pdf $$\text{Gamma}(\alpha,\lambda)\implies\frac{{\lambda}^{\alpha}}{\Gamma(\alpha)}\cdot x^{\alpha-1}e^{-\lambda x}$$, plug the value of x(=Perihelion Time) and calculate the probability.
Comments.UserDisplayName: Soumee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-06T06:06:37.533Z
Comments.Text: I want to know that how should we calculate pdfs of graphs which do not belong to the general category of pdfs such as Gaussian or Gamma distributions.
Comments.UserDisplayName: Soumee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-07T18:21:27.320Z
Comments.Text: You need to perform proper statistical tests to conclude some reasonable distributional assumptions.
Comments.UserDisplayName: naive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T01:49:36.890Z
Comments.Text: Try a larger data set.  If you dont have one use what you have and do a k fold cross validation.  If you have outliers you have outliers.  NB isn't perfect.
Comments.UserDisplayName: solarflare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T05:52:20.857Z
Comments.Text: @solarflare Actually this dataset consists of about 4000 entries. I wanted to know about the correctness and validity of the methods that I suggested.
Comments.UserDisplayName: Soumee
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-08T06:07:08.947Z
Comments.Text: 4000 may or may not be enough depending on the data
Comments.UserDisplayName: solarflare
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-11T21:04:07.403Z
Comments.Text: It seems to me your data can be well fit by a Gaussian mixture model. If you can fit each hypothesis distribution with a separate mixture model, you can do your hypothesis testing. Python has a GMM module for this -- which I would recommend tuning by hand if you can. Otherwise, they have a variational version that will find the number of mixture components automatically. It works reasonably well but requires some pre-tuning before being deployed in a script.
Comments.UserDisplayName: The Dude
Tags: <probability><probability-distribution><naive-bayes><conditional-probability>
Title: What to do when PDFs are not Gaussian/Normal in Naive Bayes Classifier
UserDisplayName: Soumee
ThreadText: # What to do when PDFs are not Gaussian/Normal in Naive Bayes Classifier
  
  (Asked by: username_0 on 2019-08-05)
  
  While analyzing the data for a given problem set, I came across a few distributions which are not Gaussian in nature. They are not even uniform or Gamma distributions(so that I can write a function, plug the parameters and calculate the "Likelihood probability" and solve it using Bayes classification method). I got a set of a few absurd looking PDFs and I am wondering how should I define them mathematically so that I can plug the parameters and calculate the likelihood probability. 
  The set of PDFs/Distributions that I got are the following and I am including some solutions that I intend to use. Please comment on their validity:
  1)
  The distribution looks like:
  $ y = ax +b $ from $ 0.8
  How to programmatically calculate
  1. The value of x where the pdf starts
  2. The value of x where the pdf ends
  3. The value of y where the pdf starts
  4. The value of y where the pdf ends
  
  However, I would have liked it better to have a generic distribution for this form of graphs so that I can plug the parameters to calculate the probability.
  2)
  This PDF looks neither uniform nor Gaussian. What kind of distribution should I consider it roughly?
  3)
  I can divide this graph into three segments. The first segment is from $2 with a steep slope, the second segment is from $3 with a moderate sope and the third segment is from $6 with a high negative slope.
  How to programmatically calculate
   1. the values of x where the graph changes its slope.
   2. the values of y where the graph changes its slope.
  
  4)
  This looks like two Gaussian densities with different mean superimposed together. But then the question arises, how do we find these two individual Gaussian densities?
  The following code may help:
  variable1=nasa1['PerihelionArg'][nasa1.PerihelionArg>190] 
  variable2=nasa1['PerihelionArg'][nasa1.PerihelionArg<190] 
  
  Find mean and variance of variable1 and variable2, find the corresponding PDFs. Define the overall PDF with a suitable range of $x$.
  5)
  This can be estimated as a Gamma distribution. We can find the mean and variance, calculate $\alpha$ and $\beta$ and finally calculate the PDF.
  It would be very helpful if someone could give their insights on the above analysis, its validity, and correctness and their suggestions regarding how problems such as these should be dealt with.
  
  **Tags:** probability, probability-distribution, naive-bayes, conditional-probability
  > **Comments:**
  > **username_0**: I am trying to train the classifier using Bayes theorem. Suppose for a given set of input, I want to determine if that asteroid is hazardous or not. For that, we need to calculate the probability P(Perihelion Time/Asteroid is Hazardous), that is, what is the probability that the asteroid takes the particular "Perihelion time"  (mentioned in the test input) given that the asteroid is "hazardous".  
  > **username_0**: So, to calculate P(Perihelion Time/Asteroid is Hazardous) we can segregate those values of Perihelion Time for which the asteroid is hazardous, compute its mean and variance and draw the pdf $$\text{Gamma}(\alpha,\lambda)\implies\frac{{\lambda}^{\alpha}}{\Gamma(\alpha)}\cdot x^{\alpha-1}e^{-\lambda x}$$, plug the value of x(=Perihelion Time) and calculate the probability.  
  > **username_0**: I want to know that how should we calculate pdfs of graphs which do not belong to the general category of pdfs such as Gaussian or Gamma distributions.  
  > **username_1**: You need to perform proper statistical tests to conclude some reasonable distributional assumptions.  
  > **username_2**: Try a larger data set.  If you dont have one use what you have and do a k fold cross validation.  If you have outliers you have outliers.  NB isn't perfect.  
  > **username_0**: @solarflare Actually this dataset consists of about 4000 entries. I wanted to know about the correctness and validity of the methods that I suggested.  
  > **username_2**: 4000 may or may not be enough depending on the data  
  > **username_3**: It seems to me your data can be well fit by a Gaussian mixture model. If you can fit each hypothesis distribution with a separate mixture model, you can do your hypothesis testing. Python has a GMM module for this -- which I would recommend tuning by hand if you can. Otherwise, they have a variational version that will find the number of mixture components automatically. It works reasonably well but requires some pre-tuning before being deployed in a script.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2019-08-20. Score: 1
  
  The relationship between the axes of graph (1) and your variables $x$ and $y$ is not clear, so this generalized answer may be helpful or useless.
  
  From graph (1) it appears that the correlation coefficient $\mathcal{C}$ of a quadratic fit of data set $\mathcal{S}$ would be much better. Consider $y\_1$ and $y\_2$ approximations of $y$.
  
  $$
  \mathcal{C} (y\_2, a, b, c, \mathcal{S}) > \mathcal{C} (y\_1, a, b, \mathcal{S}) \\
  y\_2 = ax^2 + bx + c \\
  y\_1 = ax + b
  $$
  
  To achieve a more nearly uniform distribution, perform a least squares fit for $y\_2$ against $y$ on $\mathcal{S}$ to obtain $(a, b, c)$. Then find a mapping function that produces $y'$ and use it where the uniform distribution is desired. A reasonable approximation is simply this.
  
  $$y' = \frac{y}{y\_2(x)}$$
  
  
  ---
  
  