Answers.Body: <p>Your approach will give a similar result.
  The difference with using the constraint is that the constraint allows the network to essentially confirm that it doesn't need the connections before it reallocates them (i.e. if the weight is lessened by the constraint and it results in something not working, then the weight will start increasing).</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-24T08:50:29.173Z
Answers.Text: Thanks for replying. I don't understand your last sentence (if the weight is lessened...) Could you please elaborate?
Answers.UserDisplayName: rkellerm
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-07T02:36:06.723Z
Answers.Text: I think I might have just misunderstood. I assumed that while running the S step, you were planning to add the sparsity constraint to your re-weighting function to cause low values to be less likely to increase, otherwise setting the values to 0 still allows them to grow during the S step. In my last sentence, when I referred to "constraint" I meant that.
Answers.UserDisplayName: Jeffry
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-23T16:45:53.613Z
Answers.LastActivityDate: 2017-10-23T16:45:53.613Z
Answers.UserDisplayName: Jeffry
Body: <p>I want to implement <a href="https://arxiv.org/abs/1607.04381" rel="nofollow noreferrer">DSD: Dense-Sparse-Dense training for deep neural networks</a> by Han et al. In short, the paper suggest the following training scheme to improve the network accuracy:</p>
  
  <p><a href="https://i.sstatic.net/QkuUk.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QkuUk.png" alt="enter image description here"></a></p>
  
  <ol>
  <li>Train as usual till convergence, </li>
  <li>prune the network and train with sparsity constraint, </li>
  <li>remove the sparsity constraint and let the pruned connections to recover.  </li>
  </ol>
  
  <p>My question is about step 2: train with sparsity constraint. The paper mentions training with a binary mask specifying the pruned weights to keep "untouched" so the sparsity constraint is satisfied, however that means implementing a dedicated layer that takes the binary mask as an additional blob and handles it accordingly.  </p>
  
  <p>I wonder a simpler approach will give the same result: what if after the pruning step I keep the location of the pruned weights and then use dense training, but after each iteration to zero the originally pruned weights?</p>
  
  <p>The forward path is the same, taking zero weights for the pruned weights anyway.  But would it negatively affect the backward path - since the constraint isn't there, or is it equivalent to the formal training scheme?</p>
  
  <p>Thanks.</p>
  
Tags: <deep-learning><deep-neural-networks><training>
Title: Dense-Sparse-Dense CNN training
UserDisplayName: rkellerm
ThreadText: # Dense-Sparse-Dense CNN training
  
  (Asked by: username_0 on 2017-10-23)
  
  I want to implement [DSD: Dense-Sparse-Dense training for deep neural networks](https://arxiv.org/abs/1607.04381) by Han et al. In short, the paper suggest the following training scheme to improve the network accuracy:
  
  [![enter image description here](https://i.sstatic.net/QkuUk.png)](https://i.sstatic.net/QkuUk.png)
  
  1. Train as usual till convergence,
  2. prune the network and train with sparsity constraint,
  3. remove the sparsity constraint and let the pruned connections to recover.
  
  My question is about step 2: train with sparsity constraint. The paper mentions training with a binary mask specifying the pruned weights to keep "untouched" so the sparsity constraint is satisfied, however that means implementing a dedicated layer that takes the binary mask as an additional blob and handles it accordingly.
  
  I wonder a simpler approach will give the same result: what if after the pruning step I keep the location of the pruned weights and then use dense training, but after each iteration to zero the originally pruned weights?
  
  The forward path is the same, taking zero weights for the pruned weights anyway. But would it negatively affect the backward path - since the constraint isn't there, or is it equivalent to the formal training scheme?
  
  Thanks.
  
  **Tags:** deep-learning, deep-neural-networks, training
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-10-23. Score: 1
  
  Your approach will give a similar result.
  The difference with using the constraint is that the constraint allows the network to essentially confirm that it doesn't need the connections before it reallocates them (i.e. if the weight is lessened by the constraint and it results in something not working, then the weight will start increasing).
  
  > **Comments:**
  > **username_0**: Thanks for replying. I don't understand your last sentence (if the weight is lessened...) Could you please elaborate?  
  > **username_1**: I think I might have just misunderstood. I assumed that while running the S step, you were planning to add the sparsity constraint to your re-weighting function to cause low values to be less likely to increase, otherwise setting the values to 0 still allows them to grow during the S step. In my last sentence, when I referred to "constraint" I meant that.  
  
  ---
  
  