Answers.Body: <p>I'm not sure what do you mean by one input. The input to the pruning agent is always the same, it's the convolutional layer <span class="math-container">$W$</span> of dimension <span class="math-container">$m \times h \times w$</span>. The layer is taken from baseline model that is pretrained. The input doesn't change it's always the same. The output of the pruning agent is an array of probabilities to prune a specific filter. For example if you have <span class="math-container">$3$</span> filters in a layer, the output of the pruning agent will be array of <span class="math-container">$3$</span> elements . Let's say its
  <span class="math-container">\begin{equation}
  y = [0.1, 0.6, 0.7]
  \end{equation}</span>
  Each of these elements represents probability of pruning filter <span class="math-container">$i$</span> in layer <span class="math-container">$W$</span>. So <span class="math-container">$0.1$</span> would be probability to prune filter <span class="math-container">$1$</span>, <span class="math-container">$0.6$</span> to prune filter <span class="math-container">$2$</span> and <span class="math-container">$0.7$</span> to prune filter <span class="math-container">$3$</span>.
  Let's say you sample this distribution <span class="math-container">$2$</span> times and you get: <span class="math-container">$[0, 1, 1], [0, 0, 1]$</span>. That means you would make 2 different models from the original baseline model. First model would have 2nd and 3rd filter pruned in layer <span class="math-container">$W$</span>, and second model would have 3rd filter pruned. The you run those 2 new models on your train and validation set, calculate objective function <span class="math-container">$R$</span>. Then you update parameters <span class="math-container">$\theta$</span> of your pruning agent based on <span class="math-container">$R$</span>. The original weights of layer <span class="math-container">$W$</span> stay untouched. Then you do another inference of the pruning model <span class="math-container">$\pi$</span> with updated parameters <span class="math-container">$\theta$</span> (the input is still original <span class="math-container">$W$</span>). You will get another array of probabilities and you keep repeating previous steps that i described until parameters <span class="math-container">$\theta$</span> converge. When they converge you make final pruning.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T12:55:48.410Z
Answers.Text: Thanks for your answer! but how can I get these two arrays [0,1,1],[0,0,1] ?in order to sample I normally would do something like action=model.predict(W), if I sample another time I will get the same action array, won't I ? please tell me what I'm missing.
Answers.UserDisplayName: Habib-Allah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T13:11:44.257Z
Answers.Text: That's not what sampling means, sampling translated to code (Python) is something like this : `random.random() < prob`. It's like flipping a coin. If random number is lower than probability that means you set array element to 0 (or 1 depending what you want to do).
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T13:26:03.080Z
Answers.Text: In the paper (Algorithm 1) they say sample the output distribution & then write Ai=π(θ,W). I'm confused i think. anyway thank you I will try what you said.
Answers.UserDisplayName: Habib-Allah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T13:42:03.080Z
Answers.Text: Yes, that's exactly what I said in the previous comment.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T13:51:54.160Z
Answers.Text: So this Ai=π(θ,W) means sampling the output distribution randomly ? I thought Ai=π(θ,W) means do something like Ai=agent.predict(W). Thanks :)
Answers.UserDisplayName: Habib-Allah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T14:23:55.103Z
Answers.Text: They write $\mathbf A_i \sim \pi(W, \theta)$ in Algorithm 1, this $\sim$ means they sample like I described.
Answers.UserDisplayName: Brale
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-27T11:09:31.067Z
Answers.LastActivityDate: 2020-04-27T11:18:03.067Z
Answers.LastEditDate: 2020-04-27T11:18:03.067Z
Answers.UserDisplayName: Brale
Body: <p>I was reading the paper <a href="http://arxiv.org/abs/1801.07365" rel="nofollow noreferrer">Learning to Prune Filters in Convolutional Neural Networks</a>, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input for the pruning agent (the agent is a convolutional neural network) is a <strong>2D</strong> array of shape <code>(N_l, M_l)</code>, where <code>N_l</code> is the number of filters and <code>M_l = m x h x w</code> (<code>m</code>, <code>l</code> and <code>h</code> are filter dimensions), and the output is an array of actions (each element is <strong>0</strong> (unnecessary filter) or <strong>1</strong> (necessary)) and says in order to approximate gradients we have to sample the output <code>M</code> times (using the REINFORCE algorithm).</p>
  
  <p>Since I have one input, how can I sample the output distribution multiple times (without updating the CNN parameters)? </p>
  
  <p>If I'm missing something, please, tell me where I'm wrong</p>
  
Tags: <reinforcement-learning><convolutional-neural-networks><policy-gradients><reinforce>
Title: How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?
UserDisplayName: Habib-Allah
ThreadText: # How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?
  
  (Asked by: username_0 on 2020-04-26)
  
  I was reading the paper [Learning to Prune Filters in Convolutional Neural Networks](http://arxiv.org/abs/1801.07365), which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input for the pruning agent (the agent is a convolutional neural network) is a **2D** array of shape `(N_l, M_l)`, where `N_l` is the number of filters and `M_l = m x h x w` (`m`, `l` and `h` are filter dimensions), and the output is an array of actions (each element is **0** (unnecessary filter) or **1** (necessary)) and says in order to approximate gradients we have to sample the output `M` times (using the REINFORCE algorithm).
  
  Since I have one input, how can I sample the output distribution multiple times (without updating the CNN parameters)?
  
  If I'm missing something, please, tell me where I'm wrong
  
  **Tags:** reinforcement-learning, convolutional-neural-networks, policy-gradients, reinforce
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-27. Score: 0
  
  I'm not sure what do you mean by one input. The input to the pruning agent is always the same, it's the convolutional layer $W$ of dimension $m \times h \times w$. The layer is taken from baseline model that is pretrained. The input doesn't change it's always the same. The output of the pruning agent is an array of probabilities to prune a specific filter. For example if you have $3$ filters in a layer, the output of the pruning agent will be array of $3$ elements . Let's say its
  \begin{equation}
  y = [0.1, 0.6, 0.7]
  \end{equation}
  Each of these elements represents probability of pruning filter $i$ in layer $W$. So $0.1$ would be probability to prune filter $1$, $0.6$ to prune filter $2$ and $0.7$ to prune filter $3$.
  Let's say you sample this distribution $2$ times and you get: $[0, 1, 1], [0, 0, 1]$. That means you would make 2 different models from the original baseline model. First model would have 2nd and 3rd filter pruned in layer $W$, and second model would have 3rd filter pruned. The you run those 2 new models on your train and validation set, calculate objective function $R$. Then you update parameters $\theta$ of your pruning agent based on $R$. The original weights of layer $W$ stay untouched. Then you do another inference of the pruning model $\pi$ with updated parameters $\theta$ (the input is still original $W$). You will get another array of probabilities and you keep repeating previous steps that i described until parameters $\theta$ converge. When they converge you make final pruning.
  
  > **Comments:**
  > **username_0**: Thanks for your answer! but how can I get these two arrays [0,1,1],[0,0,1] ?in order to sample I normally would do something like action=model.predict(W), if I sample another time I will get the same action array, won't I ? please tell me what I'm missing.  
  > **username_1**: That's not what sampling means, sampling translated to code (Python) is something like this : `random.random() < prob`. It's like flipping a coin. If random number is lower than probability that means you set array element to 0 (or 1 depending what you want to do).  
  > **username_0**: In the paper (Algorithm 1) they say sample the output distribution & then write Ai=π(θ,W). I'm confused i think. anyway thank you I will try what you said.  
  > **username_1**: Yes, that's exactly what I said in the previous comment.  
  > **username_0**: So this Ai=π(θ,W) means sampling the output distribution randomly ? I thought Ai=π(θ,W) means do something like Ai=agent.predict(W). Thanks :)  
  > **username_1**: They write $\mathbf A_i \sim \pi(W, \theta)$ in Algorithm 1, this $\sim$ means they sample like I described.  
  
  ---
  
  