Answers.Body: <p>I heard back from the authors of the paper.</p>
  
  <p>As expected the bernoulli sampler is non-differentiable, so as an approximation they use the expectation of the samplers gradient.  </p>
  
  <p><span class="math-container">$
  \begin{align*}  
  \frac{dL}{dv_i} &amp;= \frac{dL}{dBern(\sigma(v_i))} * \frac{dBern(\sigma(v_i))}{d\sigma(v_i)} * \frac{d\sigma(v_i)}{dv_i} \\
  &amp;\approx \frac{dL}{dBern(\sigma(v_i))} * \frac{dE[Bern(\sigma(v_i))]}{d\sigma(v_i)} * \frac{d\sigma(v_i)}{dv_i} \\
  &amp;= \frac{dL}{dBern(\sigma(v_i))} * \frac{d\sigma(v_i)}{d\sigma(v_i)} * \frac{d\sigma(v_i)}{dv_i} \\
  &amp;= \frac{dL}{dBern(\sigma(v_i))} * 1 * \frac{d\sigma(v_i)}{dv_i} \\
  \end{align*} 
  $</span></p>
  
  <p>So the answer ended up being as simple as that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-17T16:43:34.750Z
Answers.LastActivityDate: 2019-07-17T16:43:34.750Z
Answers.UserDisplayName: mshlis
Body: <p>In the paper <a href="https://arxiv.org/pdf/1905.01067.pdf" rel="nofollow noreferrer">Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask</a>, they learn a mask for the network by setting up the mask parameters as <span class="math-container">$M_i = Bern(\sigma(v_i))$</span>. Where <span class="math-container">$M$</span> is the parameter mask (<span class="math-container">$f(x;\theta, M) = f(x;M \odot \theta$</span>), <span class="math-container">$Bern$</span> is a Bernoulli sampler, <span class="math-container">$\sigma$</span> is the sigmoid function, and <span class="math-container">$v_i$</span> is some trainable parameter.</p>
  
  <p>In the paper, they learn <span class="math-container">$v_i$</span> using SGD. I was wondering how they managed to do that, because there isn't a reparameterization trick, as there is for some other distributions I see trained on in the literature (example: normal).</p>
  
Tags: <machine-learning><probability-distribution><weights>
Title: How are the parameters of the Bernoulli distribution learned?
UserDisplayName: mshlis
ThreadText: # How are the parameters of the Bernoulli distribution learned?
  
  (Asked by: username_0 on 2019-07-05)
  
  In the paper [Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask](https://arxiv.org/pdf/1905.01067.pdf), they learn a mask for the network by setting up the mask parameters as $M\_i = Bern(\sigma(v\_i))$. Where $M$ is the parameter mask ($f(x;\theta, M) = f(x;M \odot \theta$), $Bern$ is a Bernoulli sampler, $\sigma$ is the sigmoid function, and $v\_i$ is some trainable parameter.
  
  In the paper, they learn $v\_i$ using SGD. I was wondering how they managed to do that, because there isn't a reparameterization trick, as there is for some other distributions I see trained on in the literature (example: normal).
  
  **Tags:** machine-learning, probability-distribution, weights
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2019-07-17. Score: 0 | Accepted answer
  
  I heard back from the authors of the paper.
  
  As expected the bernoulli sampler is non-differentiable, so as an approximation they use the expectation of the samplers gradient.
  
  $
  \begin{align\*}
  \frac{dL}{dv\_i} &= \frac{dL}{dBern(\sigma(v\_i))} \* \frac{dBern(\sigma(v\_i))}{d\sigma(v\_i)} \* \frac{d\sigma(v\_i)}{dv\_i} \\
  &\approx \frac{dL}{dBern(\sigma(v\_i))} \* \frac{dE[Bern(\sigma(v\_i))]}{d\sigma(v\_i)} \* \frac{d\sigma(v\_i)}{dv\_i} \\
  &= \frac{dL}{dBern(\sigma(v\_i))} \* \frac{d\sigma(v\_i)}{d\sigma(v\_i)} \* \frac{d\sigma(v\_i)}{dv\_i} \\
  &= \frac{dL}{dBern(\sigma(v\_i))} \* 1 \* \frac{d\sigma(v\_i)}{dv\_i} \\
  \end{align\*}
  $
  
  So the answer ended up being as simple as that.
  
  
  ---
  
  