Answers.Body: <p><a href="https://towardsdatascience.com/why-neural-nets-can-approximate-any-function-a878768502f0" rel="nofollow noreferrer">Neural Networks can approximate any function</a>. Quoting the essence in case the article is removed in the future.</p>
  <blockquote>
  <p>The key to neural networks’ ability to approximate any function is that they incorporate non-linearity into their architecture. Each layer is associated with an activation function that applies a non-linear transformation to the output of that layer.</p>
  </blockquote>
  <p>So no, knowing the class of the probability density function is not required to approximate it via Deep Learning. With a large enough number of samples you could construct an approximation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T15:14:21.553Z
Answers.LastActivityDate: 2021-08-28T15:14:21.553Z
Answers.UserDisplayName: tnfru
Body: <p>In deep learning, models may learn the probability distribution that generated the dataset. Observe the following paragraph from <a href="https://www.deeplearningbook.org/contents/ml.html" rel="nofollow noreferrer">Chapter 5: Machine Learning Basics</a> from the book titled <a href="https://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning</a> (by Aaron Courville et al.)</p>
  <blockquote>
  <p>Unsupervised learning algorithms experience a dataset containing many
  features, then learn useful properties of the structure of this
  dataset. <strong>In the context of deep learning, we usually want to learn the
  entire probability distribution that generated a dataset, whether
  explicitly, as in density estimation</strong>, or implicitly, for tasks like
  synthesis or denoising. Some other unsupervised learning algorithms
  perform other roles, like clustering, which consists of dividing the
  dataset into clusters of similar examples.</p>
  </blockquote>
  <p>I read about density estimation in the same chapter, as given below</p>
  <blockquote>
  <p>In the density estimation problem, the machine learning algorithm is
  asked to learn a function <span class="math-container">$p_{model} : R^n \rightarrow R$</span>, where
  <span class="math-container">$p_{model}$</span>(x) <strong>can be interpreted as a probability density function
  (if <span class="math-container">$x$</span> is continuous)</strong> or a probability mass function (if <span class="math-container">$x$</span> is
  discrete) on the space that the examples were drawn from.</p>
  </blockquote>
  <p>This question is focused on explicit probability density estimation in continuous case i.e., learning density function <span class="math-container">$p_{model}$</span>  directly.</p>
  <p>Suppose I have a dataset <span class="math-container">$D$</span> with <span class="math-container">$n$</span> continuous random variables (features) <span class="math-container">$X_1, X_2, X_3,\cdots, X_n$</span>. And I don't know anything about the probability density function of individual random variables. That is, I don't know about any information about any <span class="math-container">$X_i$</span> , such as, whether <span class="math-container">$X_i$</span> follows normal distribution or any other distribution. Then, is it possible to learn density function explicitly? Or do I need to provide some necessary information such as the class of probability distribution function to be learned?</p>
  <p>I am thinking as follows:</p>
  <p>If I have some information about <span class="math-container">$X_i$</span>, such as: <span class="math-container">$X_i$</span> falls to a well known distribution, then I can learn the parameters of the underlying density function from <span class="math-container">$D$</span>. So, is it mandatory to know some information about the underlying probability density function.</p>
  
Tags: <density-estimation>
Title: Is knowing the class of probability density function mandatory for explicit density estimation?
UserDisplayName: hanugm
ThreadText: # Is knowing the class of probability density function mandatory for explicit density estimation?
  
  (Asked by: username_0 on 2021-08-28)
  
  In deep learning, models may learn the probability distribution that generated the dataset. Observe the following paragraph from [Chapter 5: Machine Learning Basics](https://www.deeplearningbook.org/contents/ml.html) from the book titled [Deep Learning](https://www.deeplearningbook.org/) (by Aaron Courville et al.)
  
  > Unsupervised learning algorithms experience a dataset containing many
  > features, then learn useful properties of the structure of this
  > dataset. **In the context of deep learning, we usually want to learn the
  > entire probability distribution that generated a dataset, whether
  > explicitly, as in density estimation**, or implicitly, for tasks like
  > synthesis or denoising. Some other unsupervised learning algorithms
  > perform other roles, like clustering, which consists of dividing the
  > dataset into clusters of similar examples.
  
  I read about density estimation in the same chapter, as given below
  
  > In the density estimation problem, the machine learning algorithm is
  > asked to learn a function $p\_{model} : R^n \rightarrow R$, where
  > $p\_{model}$(x) **can be interpreted as a probability density function
  > (if $x$ is continuous)** or a probability mass function (if $x$ is
  > discrete) on the space that the examples were drawn from.
  
  This question is focused on explicit probability density estimation in continuous case i.e., learning density function $p\_{model}$ directly.
  
  Suppose I have a dataset $D$ with $n$ continuous random variables (features) $X\_1, X\_2, X\_3,\cdots, X\_n$. And I don't know anything about the probability density function of individual random variables. That is, I don't know about any information about any $X\_i$ , such as, whether $X\_i$ follows normal distribution or any other distribution. Then, is it possible to learn density function explicitly? Or do I need to provide some necessary information such as the class of probability distribution function to be learned?
  
  I am thinking as follows:
  
  If I have some information about $X\_i$, such as: $X\_i$ falls to a well known distribution, then I can learn the parameters of the underlying density function from $D$. So, is it mandatory to know some information about the underlying probability density function.
  
  **Tags:** density-estimation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-28. Score: 0
  
  [Neural Networks can approximate any function](https://towardsdatascience.com/why-neural-nets-can-approximate-any-function-a878768502f0). Quoting the essence in case the article is removed in the future.
  
  > The key to neural networks’ ability to approximate any function is that they incorporate non-linearity into their architecture. Each layer is associated with an activation function that applies a non-linear transformation to the output of that layer.
  
  So no, knowing the class of the probability density function is not required to approximate it via Deep Learning. With a large enough number of samples you could construct an approximation.
  
  
  ---
  
  