Answers.Body: <p>The probabilistic models that represent distributions <em>implicitly</em> are, for example, the <a href="https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf" rel="nofollow noreferrer">GANs</a>. (Goodfellow is one of the authors of the original <a href="https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf" rel="nofollow noreferrer">GAN</a> model).</p>
  <p>In the paper <a href="https://arxiv.org/pdf/1702.08235.pdf" rel="nofollow noreferrer">Variational Inference using Implicit Distributions</a> (2017), the authors write</p>
  <blockquote>
  <p>Implicit distributions are probability models whose <strong>probability density function may be intractable</strong>, but there is a way to</p>
  <ol>
  <li><p><strong>sample</strong> from them exactly and/or calculate and <strong>approximate expectations</strong> under them, and</p>
  </li>
  <li><p>calculate or <strong>estimate gradients</strong> of such expectations with respect to model parameters.</p>
  </li>
  </ol>
  <p>Implicit models have been successfully applied to generative modelling in generative adversarial networks (GANs)</p>
  <p>A popular example of implicit models are stochastic generative networks: samples from a simple distribution - such as uniform or Gaussian - are transformed nonlinearly and non-invertably by a deep neural network</p>
  </blockquote>
  <p>They also provide a table (table 1) that shows some probabilistic models that use an implicit distribution. Here <span class="math-container">$I$</span> denotes inference only. VI stands for variational inference.</p>
  <p><a href="https://i.sstatic.net/KvzOw.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KvzOw.png" alt="enter image description here" /></a></p>
  <p>See this blog post <a href="https://www.inference.vc/variational-inference-with-implicit-probabilistic-models-part-1-2/" rel="nofollow noreferrer">Variational Inference using Implicit Models, Part I: Bayesian Logistic Regression</a>. See also the paper <a href="https://arxiv.org/pdf/1610.03483.pdf" rel="nofollow noreferrer">Learning in Implicit Generative Models</a> (2017).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T18:41:54.553Z
Answers.Text: Thanks for the answer. Do you mind briefly clarifying what is meant by "are transformed nonlinearly and non-invertably by a deep neural network" for a machine learning novice such as myself? I was actually recently studying exercise 1.4. of *Pattern Recognition and Machine Learning* by Christopher Bishop, which was an exercise on the linear/non-linear transformation of a density function, so I'm curious about this.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T18:42:50.287Z
Answers.Text: @ThePointer Neural networks **non**-linearly transform their inputs because neural networks (usually) use **non**-linear activation functions (i.e. functions that are non-linear, such as the **sigmoid** function).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T18:43:52.800Z
Answers.Text: Wait, neural networks *linearly* transform their inputs because they use *non-linear* activation functions? Is that correct is there a typo here?
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T18:44:14.733Z
Answers.Text: @ThePointer It was a typo.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T18:44:40.727Z
Answers.Text: Ok. Thank you for taking the time to post this answer.
Answers.UserDisplayName: The Pointer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-29T03:55:31.463Z
Answers.Text: @DuttaA Well, I'm listing other models/algorithms that make use of implicit distributions (table 1 of the first paper), so I don't think I am focusing on the GAN. I just mentioned the GAN because Goodfellow was probably referring to it. To be honest, I wasn't aware of this terminology before this question came up. In fact, if you see the edit history of this answer, I initially said that the VAE also represented distributions implicitly, while this isn't apparently the case according to some papers, because I thought they were referring to the variational distribution.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-28T18:02:47.790Z
Answers.LastActivityDate: 2020-02-28T18:38:29.150Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: nbro
Body: <p>I am currently studying Deep Learning by Goodfellow, Bengio, and Courville. In chapter <strong>5.1.2 The Performance Measure,</strong> <em>P</em>, the authors say the following:</p>
  <blockquote>
  <p>The choice of performance measure may seem straightforward and objective, but it is often difficult to choose a performance measure that corresponds well to the desired behavior of the system.</p>
  <p>In some cases, this is because it is difficult to decide what should be measured. For example, when performing a transcription task, should we measure the accuracy  of the system at transcribing entire sequences, or should we use a more fine-grained performance measure that gives partial credit for getting some elements of the sequence correct? When performing a regression task, should we penalize the  system more if it frequently makes medium-sized mistakes or if it rarely makes very large mistakes? These kinds of design choices depend on the application.</p>
  <p>In other cases, we know what quantity we would ideally like to measure, but measuring it is impractical. For example, this arises frequently in the context of density estimation. Many of the best probabilistic models represent probability distributions only implicitly. Computing the actual probability value assigned to a  specific point in space in many such models is intractable. In these cases, one must design an alternative criterion that still corresponds to the design objectives, or design a good approximation to the desired criterion.</p>
  </blockquote>
  <p>It is this part that interests me:</p>
  <blockquote>
  <p>Many of the best probabilistic models represent probability distributions only implicitly.</p>
  </blockquote>
  <p>I don't have the experience to understand what this means (what does it mean to represent distributions &quot;implicitly&quot;?). I would greatly appreciate it if people would please take the time to elaborate upon this.</p>
  
Tags: <deep-learning><probability-distribution>
Title: Many of the best probabilistic models represent probability distributions only implicitly
UserDisplayName: The Pointer
ThreadText: # Many of the best probabilistic models represent probability distributions only implicitly
  
  (Asked by: username_0 on 2020-02-25)
  
  I am currently studying Deep Learning by Goodfellow, Bengio, and Courville. In chapter **5.1.2 The Performance Measure,** *P*, the authors say the following:
  
  > The choice of performance measure may seem straightforward and objective, but it is often difficult to choose a performance measure that corresponds well to the desired behavior of the system.
  >
  > In some cases, this is because it is difficult to decide what should be measured. For example, when performing a transcription task, should we measure the accuracy of the system at transcribing entire sequences, or should we use a more fine-grained performance measure that gives partial credit for getting some elements of the sequence correct? When performing a regression task, should we penalize the system more if it frequently makes medium-sized mistakes or if it rarely makes very large mistakes? These kinds of design choices depend on the application.
  >
  > In other cases, we know what quantity we would ideally like to measure, but measuring it is impractical. For example, this arises frequently in the context of density estimation. Many of the best probabilistic models represent probability distributions only implicitly. Computing the actual probability value assigned to a specific point in space in many such models is intractable. In these cases, one must design an alternative criterion that still corresponds to the design objectives, or design a good approximation to the desired criterion.
  
  It is this part that interests me:
  
  > Many of the best probabilistic models represent probability distributions only implicitly.
  
  I don't have the experience to understand what this means (what does it mean to represent distributions "implicitly"?). I would greatly appreciate it if people would please take the time to elaborate upon this.
  
  **Tags:** deep-learning, probability-distribution
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-02-28. Score: 2 | Accepted answer
  
  The probabilistic models that represent distributions *implicitly* are, for example, the [GANs](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf). (Goodfellow is one of the authors of the original [GAN](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf) model).
  
  In the paper [Variational Inference using Implicit Distributions](https://arxiv.org/pdf/1702.08235.pdf) (2017), the authors write
  
  > Implicit distributions are probability models whose **probability density function may be intractable**, but there is a way to
  >
  > 1. **sample** from them exactly and/or calculate and **approximate expectations** under them, and
  > 2. calculate or **estimate gradients** of such expectations with respect to model parameters.
  >
  > Implicit models have been successfully applied to generative modelling in generative adversarial networks (GANs)
  >
  > A popular example of implicit models are stochastic generative networks: samples from a simple distribution - such as uniform or Gaussian - are transformed nonlinearly and non-invertably by a deep neural network
  
  They also provide a table (table 1) that shows some probabilistic models that use an implicit distribution. Here $I$ denotes inference only. VI stands for variational inference.
  
  [![enter image description here](https://i.sstatic.net/KvzOw.png)](https://i.sstatic.net/KvzOw.png)
  
  See this blog post [Variational Inference using Implicit Models, Part I: Bayesian Logistic Regression](https://www.inference.vc/variational-inference-with-implicit-probabilistic-models-part-1-2/). See also the paper [Learning in Implicit Generative Models](https://arxiv.org/pdf/1610.03483.pdf) (2017).
  
  > **Comments:**
  > **username_0**: Thanks for the answer. Do you mind briefly clarifying what is meant by "are transformed nonlinearly and non-invertably by a deep neural network" for a machine learning novice such as myself? I was actually recently studying exercise 1.4. of *Pattern Recognition and Machine Learning* by Christopher Bishop, which was an exercise on the linear/non-linear transformation of a density function, so I'm curious about this.  
  > **username_1**: @ThePointer Neural networks **non**-linearly transform their inputs because neural networks (usually) use **non**-linear activation functions (i.e. functions that are non-linear, such as the **sigmoid** function).  
  > **username_0**: Wait, neural networks *linearly* transform their inputs because they use *non-linear* activation functions? Is that correct is there a typo here?  
  > **username_1**: @ThePointer It was a typo.  
  > **username_0**: Ok. Thank you for taking the time to post this answer.  
  > **username_1**: @DuttaA Well, I'm listing other models/algorithms that make use of implicit distributions (table 1 of the first paper), so I don't think I am focusing on the GAN. I just mentioned the GAN because Goodfellow was probably referring to it. To be honest, I wasn't aware of this terminology before this question came up. In fact, if you see the edit history of this answer, I initially said that the VAE also represented distributions implicitly, while this isn't apparently the case according to some papers, because I thought they were referring to the variational distribution.  
  
  ---
  
  