Body: <p>I've been playing around with pruning neural networks. One interesting thing I've found is that pruning the weights with the lowest L1-norm <em>along a specific dimension</em> seems to give better results than simply pruning all of the weights with the lowest L1-norm (which I believe is the standard method; for example this is what <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.l1_unstructured.html#torch-nn-utils-prune-l1-unstructured" rel="nofollow noreferrer">torch.nn.utils.prune.l1_unstructured</a> does).</p>
  <p>Does anyone have an explanation for why this might be, or knows of any research in this area?</p>
  <p>I'm aware that structured pruning removes entire channels in a specific dimension. But I'm referring to unstructured pruning here, where I remove a subset of the weights along a specific dimension.</p>
  <p>Admittedly I've only done very limited benchmarking of this. See <a href="https://github.com/nfergu/l1_dim_pruner" rel="nofollow noreferrer">this repo</a> for my implementation, and some benchmark details.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-29T18:20:12.617Z
Comments.Text: Please, put more contents related to "the lowest L1-norm along a specific dimension". Especially, specific dimension. Also is there any other study in this method?
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-30T09:19:37.023Z
Comments.Text: @CloudCho What I mean specifically is, pruning along dimension 1 (the second dimension) of each of the full-connected layers of the network. Specifically, I use torch.topk with dim=1 to find the values with the lowest L1-norm sorted along dimension 1. The implementation and benchmark results are here: https://github.com/nfergu/l1_dim_pruner/
Comments.UserDisplayName: Neil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-01T01:52:14.373Z
Comments.Text: Have you tried to perform pruning in other dimension?
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-02T11:23:49.233Z
Comments.Text: @CloudCho Yes, I've tried pruning along dimension 0 (the first dimension). Performance is not as good (worse than standard L1-norm pruning).
Comments.UserDisplayName: Neil
Tags: <neural-networks>
Title: Pruning a specific dimension in a neural network using L1-norm
UserDisplayName: Neil
ThreadText: # Pruning a specific dimension in a neural network using L1-norm
  
  (Asked by: username_0 on 2023-09-29)
  
  I've been playing around with pruning neural networks. One interesting thing I've found is that pruning the weights with the lowest L1-norm *along a specific dimension* seems to give better results than simply pruning all of the weights with the lowest L1-norm (which I believe is the standard method; for example this is what [torch.nn.utils.prune.l1\_unstructured](https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.l1_unstructured.html#torch-nn-utils-prune-l1-unstructured) does).
  
  Does anyone have an explanation for why this might be, or knows of any research in this area?
  
  I'm aware that structured pruning removes entire channels in a specific dimension. But I'm referring to unstructured pruning here, where I remove a subset of the weights along a specific dimension.
  
  Admittedly I've only done very limited benchmarking of this. See [this repo](https://github.com/nfergu/l1_dim_pruner) for my implementation, and some benchmark details.
  
  **Tags:** neural-networks
  > **Comments:**
  > **username_1**: Please, put more contents related to "the lowest L1-norm along a specific dimension". Especially, specific dimension. Also is there any other study in this method?  
  > **username_0**: @CloudCho What I mean specifically is, pruning along dimension 1 (the second dimension) of each of the full-connected layers of the network. Specifically, I use torch.topk with dim=1 to find the values with the lowest L1-norm sorted along dimension 1. The implementation and benchmark results are here: https://github.com/nfergu/l1_dim_pruner/  
  > **username_1**: Have you tried to perform pruning in other dimension?  
  > **username_0**: @CloudCho Yes, I've tried pruning along dimension 0 (the first dimension). Performance is not as good (worse than standard L1-norm pruning).  
  
  ---
  
  ## 0 answers
  
  