Answers.Body: <p>I wouldn't call it nor attention nor dynamic convolution.</p>
  <p>Reason being that everything is static. if for conv(x) you refer to a standard convolution then that would imply a static kernel, so nothing fancy going on there but just a classic multichannel CNN, and adding 3 learnable parameters is basically just adding a linear layer (not dense) on top of those features. So in inference phase one of those parameters will be higher than the others and the features coming from the convolution, associated with that parameter, will overcome the others. Which doesn't really sound like attention.</p>
  <p>The closest paper to what you're suggesting I can think about is:
  <a href="https://arxiv.org/pdf/1907.10830.pdf" rel="nofollow noreferrer">U-GAT-IT: Unsupervised generative attentional notworks with adaptive layer instance normalization for image-to-image translation.</a></p>
  <p>You can see in the image below that in their generator the authors apply individual weights to each feature map, but the crucial difference is that these weights are not just extra initialized parameters, they come from an auxiliary classifier trained precisely to generate attention masks, idea taken from <a href="https://arxiv.org/pdf/1512.04150.pdf" rel="nofollow noreferrer">Learning Deep Features for Discriminative Localization</a> (from which I took the second image).</p>
  <p><a href="https://i.sstatic.net/Lrd7b.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Lrd7b.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/Y0c7T.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Y0c7T.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-08T12:32:16.860Z
Answers.LastActivityDate: 2022-03-08T12:32:16.860Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>See <a href="https://arxiv.org/abs/1912.03458" rel="nofollow noreferrer">Dynamic Convolution: Attention over Convolution Kernels</a> by Yinpeng Chen et al.</p>
  <p>The convolution kernels are generated by taking a weighted average of <code>K=4</code> kernels. The weights are determined <em>non-linearly</em> via channel attention (i.e. &quot;excitation&quot; in SE networks) that uses global average pooling and a dense network.</p>
  <p><a href="https://i.sstatic.net/iQEwF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/iQEwF.png" alt="Dynamic Convolution: Attention over Convolution Kernels - Figure 2" /></a></p>
  <p>The paper also discusses some &quot;training tricks&quot;, namely by limiting the space of possible weights via a softmax with &quot;temperature&quot; of <code>T=30</code> to soften the max further.</p>
  <p>See also: an <a href="https://github.com/kaijieshi7/Dynamic-convolution-Pytorch/blob/4befa50c97de72cd093316edb29522e8ebd8fc5e/dynamic_conv.py#L100-L190" rel="nofollow noreferrer">implementation in PyTorch</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-27T08:01:48.770Z
Answers.LastActivityDate: 2022-09-27T08:07:45.200Z
Answers.LastEditDate: 2022-09-27T08:07:45.200Z
Answers.UserDisplayName: Mateen Ulhaq
Body: <p>Are there any examples of people performing multiple convolutions at a single depth and then performing feature max aggregation as a convex combination as a form of &quot;dynamic convolutions&quot;?</p>
  <p>To be more precise: Say you have an input x, and you generate</p>
  <pre><code>Y_1 = conv(x) 
  Y_2 = conv(x)
  Y_3 = conv(x)
  
  Y = torch.cat([Y_1,Y_2,Y_3]) 
  Weights = nn.Parameter(torch.rand(1,3)) 
  Weights_normalized = nn.softmax(weights) 
  Attended_features = torch.matmul(Y, weights_normalized.t())
  </code></pre>
  <p>So, essentially, you are learning a weighting of the feature maps through this averaging procedure.</p>
  <p>Some of you may be familiar with the &quot;Dynamic Convolutions&quot; paper. I’m just curious if you all would consider this dynamic convolution or attention of feature maps. Have you seen it before?</p>
  <p>If the code isn’t clear, this is just taking an optimized linear combination of the convolution algorithm feature maps.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-09T09:53:49.367Z
Comments.Text: Is the paper you're referring to [Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference](https://arxiv.org/abs/1912.03203)?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-09T18:26:47.377Z
Comments.Text: @nbro exactly the paper I am referencing but they are aggregating on the  weights of the conv layer.
Comments.UserDisplayName: ADA
Tags: <neural-networks><deep-learning><convolutional-neural-networks><attention><convolutional-layers>
Title: Is this aggregation of multiple convolutions of the same input a type of attention or dynamic convolution?
UserDisplayName: ADA
ThreadText: # Is this aggregation of multiple convolutions of the same input a type of attention or dynamic convolution?
  
  (Asked by: username_0 on 2022-03-06)
  
  Are there any examples of people performing multiple convolutions at a single depth and then performing feature max aggregation as a convex combination as a form of "dynamic convolutions"?
  
  To be more precise: Say you have an input x, and you generate
  
  ```
  Y_1 = conv(x) 
  Y_2 = conv(x)
  Y_3 = conv(x)
  
  Y = torch.cat([Y_1,Y_2,Y_3]) 
  Weights = nn.Parameter(torch.rand(1,3)) 
  Weights_normalized = nn.softmax(weights) 
  Attended_features = torch.matmul(Y, weights_normalized.t())
  
  ```
  
  So, essentially, you are learning a weighting of the feature maps through this averaging procedure.
  
  Some of you may be familiar with the "Dynamic Convolutions" paper. I’m just curious if you all would consider this dynamic convolution or attention of feature maps. Have you seen it before?
  
  If the code isn’t clear, this is just taking an optimized linear combination of the convolution algorithm feature maps.
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, attention, convolutional-layers
  
  > **Comments:**
  > **username_1**: Is the paper you're referring to [Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference](https://arxiv.org/abs/1912.03203)?  
  > **username_0**: @username_1 exactly the paper I am referencing but they are aggregating on the  weights of the conv layer.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-03-08. Score: 1
  
  I wouldn't call it nor attention nor dynamic convolution.
  
  Reason being that everything is static. if for conv(x) you refer to a standard convolution then that would imply a static kernel, so nothing fancy going on there but just a classic multichannel CNN, and adding 3 learnable parameters is basically just adding a linear layer (not dense) on top of those features. So in inference phase one of those parameters will be higher than the others and the features coming from the convolution, associated with that parameter, will overcome the others. Which doesn't really sound like attention.
  
  The closest paper to what you're suggesting I can think about is:
  [U-GAT-IT: Unsupervised generative attentional notworks with adaptive layer instance normalization for image-to-image translation.](https://arxiv.org/pdf/1907.10830.pdf)
  
  You can see in the image below that in their generator the authors apply individual weights to each feature map, but the crucial difference is that these weights are not just extra initialized parameters, they come from an auxiliary classifier trained precisely to generate attention masks, idea taken from [Learning Deep Features for Discriminative Localization](https://arxiv.org/pdf/1512.04150.pdf) (from which I took the second image).
  
  [![enter image description here](https://i.sstatic.net/Lrd7b.png)](https://i.sstatic.net/Lrd7b.png)
  
  [![enter image description here](https://i.sstatic.net/Y0c7T.png)](https://i.sstatic.net/Y0c7T.png)
  
  
  ---
  
  ### Answer by username_3 on 2022-09-27. Score: 0
  
  See [Dynamic Convolution: Attention over Convolution Kernels](https://arxiv.org/abs/1912.03458) by Yinpeng Chen et al.
  
  The convolution kernels are generated by taking a weighted average of `K=4` kernels. The weights are determined *non-linearly* via channel attention (i.e. "excitation" in SE networks) that uses global average pooling and a dense network.
  
  [![Dynamic Convolution: Attention over Convolution Kernels - Figure 2](https://i.sstatic.net/iQEwF.png)](https://i.sstatic.net/iQEwF.png)
  
  The paper also discusses some "training tricks", namely by limiting the space of possible weights via a softmax with "temperature" of `T=30` to soften the max further.
  
  See also: an [implementation in PyTorch](https://github.com/kaijieshi7/Dynamic-convolution-Pytorch/blob/4befa50c97de72cd093316edb29522e8ebd8fc5e/dynamic_conv.py#L100-L190).
  
  
  ---
  
  