Answers.Body: <p>The <em>reparameterization trick</em> (also known as the <em>pathwise derivative</em> or <em>infinitesimal perturbation analysis</em>) is a method for calculating the gradient of a function of a random variable. It is used, for example, in variational autoencoders or deterministic policy gradient algorithms.</p>
  <p>If you plan on working with models that involve random variables, you definitely need to understand what the reparameterization trick is.</p>
  <p>You will also need to understand the other method to calculate gradients for functions of random variables, which is known as the <em>likelihood ratio</em> (also known as the <em>score function</em> or  <em>the REINFORCE gradient</em>).</p>
  <p>If your definition of a &quot;traditional&quot; neural network does not involve random variables, then such a method is irrelevant.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-19T11:34:11.317Z
Answers.Text: Note that the re-parametrization trick is used to reduce the variance (i.e. it's a _variance reduction technique_). The authors of the VAE state this in the VAE paper (if I remember correctly). So, is the re-parametrization really needed or only needed to reduce the variance? Moreover, are you sure that "REINFORCE gradient" is generally a synonym for "likelihood ratio"? Maybe in the context of reinforcement learning and the REINFORCE algorithm, but isn't the likelihood ratio something more general?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-19T16:22:47.043Z
Answers.Text: Yes, REINFORCE gradient definitely means the same thing as score function or likelihood ratio (at least when the latter two are talking about gradient estimation).
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-19T16:27:41.520Z
Answers.Text: The reparameterization trick has lower variance than likelihood ratios. I guess you could call it a variance reduction technique in this sense (assuming that either method can be applied), but typically variance reduction is referring to the use of some sort of control variate.
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T12:40:49.083Z
Answers.Text: I agree with @nbro, REINFORCE gradient is not synonymous with the likelihood ratio. The likelihood ratio is something extremely general from statistics that has been around much longer than REINFORCE.
Answers.UserDisplayName: David
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T15:47:26.943Z
Answers.Text: https://arxiv.org/abs/1506.05254 page 3, nomenclature, "What we call the score function
  estimator (via [3]) is alternatively called the likelihood ratio estimator [5] and REINFORCE [26]."
Answers.UserDisplayName: Taw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T16:42:14.990Z
Answers.Text: @Taw The point is: [REINFORCE is a **reinforcement learning** algorithm](http://incompleteideas.net/book/RLbook2020.pdf#page=348), while the term "score function" is used more generally. Do you understand what we want to say? The score function appears e.g. in variational inference too.  Saying that REINFORCE is the score function is similar to saying that gradient descent is the chain rule. No. Gradient descent is an algorithm that uses a specific version of the chain rule applied to neural nets. So, REINFORCE is an algorithm, it's not a score function. It uses a score function.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T16:48:28.203Z
Answers.Text: See [my answer](https://ai.stackexchange.com/a/33829/2444) here, for instance, and the linked paper about Monte Carlo gradient estimation.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-17T10:45:34.363Z
Answers.LastActivityDate: 2021-11-19T11:36:35.207Z
Answers.LastEditDate: 2021-11-19T11:36:35.207Z
Answers.UserDisplayName: Taw
Answers.Body: <p>Yes, the reparametrization trick can be useful in the context of <a href="http://proceedings.mlr.press/v37/blundell15.pdf" rel="nofollow noreferrer"><strong>variational Bayesian neural networks</strong></a>, although other more effective variance reduction techniques are more commonly used (in particular, <a href="https://arxiv.org/abs/1803.04386" rel="nofollow noreferrer">the flipout estimator</a>). See <a href="https://github.com/tensorflow/probability/blob/main/tensorflow_probability/examples/bayesian_neural_network.py" rel="nofollow noreferrer">this implementation</a> of BNNs that uses Flipout, but TensorFlow Probability, the library used to implement that example, also <a href="https://www.tensorflow.org/probability/api_docs/python/tfp/layers/Convolution2DReparameterization" rel="nofollow noreferrer">provides layers that implement the reparametrization trick</a>.</p>
  <p>Note that the reparametrization trick is used in the context of <a href="https://arxiv.org/abs/1312.6114" rel="nofollow noreferrer"><strong>variational</strong> auto-encoders (VAEs)</a> (so not in the context of deterministic auto-encoders). VAEs and BNNs have a lot in common: both are based on stochastic variational inference (i.e. variational inference combined with stochastic gradient descent). So, whenever you have some sampling or some stochastic operation, the reparametrization trick could turn out to be useful. However, right now, I am only familiar with these two types of models that use it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-19T17:09:00.733Z
Answers.LastActivityDate: 2021-12-19T17:09:00.733Z
Answers.UserDisplayName: nbro
Body: <p>I came across the  &quot;reparametrization trick&quot; for the first time in the <a href="https://mml-book.github.io/book/mml-book.pdf#page=158" rel="nofollow noreferrer">following paragraph</a> from the chapter named <em>Vector Calculus</em> from the test book titled <strong><a href="https://mml-book.github.io/book/mml-book.pdf" rel="nofollow noreferrer">Mathematics for Machine Learning</a></strong> by <em>Marc Peter Deisenroth et al.</em></p>
  <blockquote>
  <p>The Jacobian determinant and variable transformations will become
  relevant ... when we transform random variables and probability
  distributions. These transformations are extremely relevant in machine
  learning <strong>in the context of training deep neural networks using the
  reparametrization trick, also called infinite perturbation analysis</strong>.</p>
  </blockquote>
  <p>The trick has been used in the context of neural networks training in the quoted paragraph. But when I search about the reparametrization trick, I found it only or widely in training autoencoders.</p>
  <p>In the context of training a traditional deep neural network, is the trick useful?</p>
  
Tags: <neural-networks><training><autoencoders>
Title: Can I apply reparametrization trick on "any" deep neural network?
UserDisplayName: hanugm
ThreadText: # Can I apply reparametrization trick on "any" deep neural network?
  
  (Asked by: username_0 on 2021-11-16)
  
  I came across the "reparametrization trick" for the first time in the [following paragraph](https://mml-book.github.io/book/mml-book.pdf#page=158) from the chapter named *Vector Calculus* from the test book titled **[Mathematics for Machine Learning](https://mml-book.github.io/book/mml-book.pdf)** by *Marc Peter Deisenroth et al.*
  
  > The Jacobian determinant and variable transformations will become
  > relevant ... when we transform random variables and probability
  > distributions. These transformations are extremely relevant in machine
  > learning **in the context of training deep neural networks using the
  > reparametrization trick, also called infinite perturbation analysis**.
  
  The trick has been used in the context of neural networks training in the quoted paragraph. But when I search about the reparametrization trick, I found it only or widely in training autoencoders.
  
  In the context of training a traditional deep neural network, is the trick useful?
  
  **Tags:** neural-networks, training, autoencoders
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-11-17. Score: 2
  
  The *reparameterization trick* (also known as the *pathwise derivative* or *infinitesimal perturbation analysis*) is a method for calculating the gradient of a function of a random variable. It is used, for example, in variational autoencoders or deterministic policy gradient algorithms.
  
  If you plan on working with models that involve random variables, you definitely need to understand what the reparameterization trick is.
  
  You will also need to understand the other method to calculate gradients for functions of random variables, which is known as the *likelihood ratio* (also known as the *score function* or *the REINFORCE gradient*).
  
  If your definition of a "traditional" neural network does not involve random variables, then such a method is irrelevant.
  
  > **Comments:**
  > **username_2**: Note that the re-parametrization trick is used to reduce the variance (i.e. it's a _variance reduction technique_). The authors of the VAE state this in the VAE paper (if I remember correctly). So, is the re-parametrization really needed or only needed to reduce the variance? Moreover, are you sure that "REINFORCE gradient" is generally a synonym for "likelihood ratio"? Maybe in the context of reinforcement learning and the REINFORCE algorithm, but isn't the likelihood ratio something more general?  
  > **username_1**: Yes, REINFORCE gradient definitely means the same thing as score function or likelihood ratio (at least when the latter two are talking about gradient estimation).  
  > **username_1**: The reparameterization trick has lower variance than likelihood ratios. I guess you could call it a variance reduction technique in this sense (assuming that either method can be applied), but typically variance reduction is referring to the use of some sort of control variate.  
  > **username_3**: I agree with @nbro, REINFORCE gradient is not synonymous with the likelihood ratio. The likelihood ratio is something extremely general from statistics that has been around much longer than REINFORCE.  
  > **username_1**: https://arxiv.org/abs/1506.05254 page 3, nomenclature, "What we call the score function estimator (via [3]) is alternatively called the likelihood ratio estimator [5] and REINFORCE [26]."  
  > **username_2**: @Taw The point is: [REINFORCE is a **reinforcement learning** algorithm](http://incompleteideas.net/book/RLbook2020.pdf#page=348), while the term "score function" is used more generally. Do you understand what we want to say? The score function appears e.g. in variational inference too.  Saying that REINFORCE is the score function is similar to saying that gradient descent is the chain rule. No. Gradient descent is an algorithm that uses a specific version of the chain rule applied to neural nets. So, REINFORCE is an algorithm, it's not a score function. It uses a score function.  
  > **username_2**: See [my answer](https://ai.stackexchange.com/a/33829/2444) here, for instance, and the linked paper about Monte Carlo gradient estimation.  
  
  ---
  
  ### Answer by username_2 on 2021-12-19. Score: 0
  
  Yes, the reparametrization trick can be useful in the context of [**variational Bayesian neural networks**](http://proceedings.mlr.press/v37/blundell15.pdf), although other more effective variance reduction techniques are more commonly used (in particular, [the flipout estimator](https://arxiv.org/abs/1803.04386)). See [this implementation](https://github.com/tensorflow/probability/blob/main/tensorflow_probability/examples/bayesian_neural_network.py) of BNNs that uses Flipout, but TensorFlow Probability, the library used to implement that example, also [provides layers that implement the reparametrization trick](https://www.tensorflow.org/probability/api_docs/python/tfp/layers/Convolution2DReparameterization).
  
  Note that the reparametrization trick is used in the context of [**variational** auto-encoders (VAEs)](https://arxiv.org/abs/1312.6114) (so not in the context of deterministic auto-encoders). VAEs and BNNs have a lot in common: both are based on stochastic variational inference (i.e. variational inference combined with stochastic gradient descent). So, whenever you have some sampling or some stochastic operation, the reparametrization trick could turn out to be useful. However, right now, I am only familiar with these two types of models that use it.
  
  
  ---
  
  