Body: <p>Positional encoding (PE) is an essential part of the self-attention layers in the transformer architectures since without adding it in some way (fixed of learnable) to the input embeddings model has ultimately no notion of order and is permutationally equivariant and the given token attends to the far separate and local tokens identically.</p>
  <p>The convolution operation with a local filter, say of size <span class="math-container">$3, 5$</span> for 1D convolutions or <span class="math-container">$3 \times 3, 5 \times 5$</span> for 2D convolutions, has some notion of locality by construction. However, within this neighborhood, all pixels are treated in the same way.</p>
  <p>However, it may be the case, that it is important, that the given pixel is the central for the application of this filter, whereas the other is close to the boundary. For small filters
  <span class="math-container">$3 \times 3$</span> it is probably not an issue, but for the larger - injection of PE can be useful.</p>
  <p>Has this question been investigated in the literature? Are there any architectures with PE + convolutions?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-01T21:07:03.197Z
Comments.Text: "However, within this neighborhood, all pixels are treated in the same way.", this is not true (unless I'm misinterpreting this statement): that's the point of learning the values of the kernel.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-04T06:20:10.003Z
Comments.Text: @nbro well, probably my question is really ill-posed, but I mean, that in the 1D conv for example we have a stuff like $w_{-1} x_{i - 1} + w_{0} x_{i} + w_{1} x_{i + 1}$, and here none of the weights and locations $w_{j}, x_{i-j}$ is preferred over the other. However, i would think, that for long enough filters at least, values in the middle are more important, that on the edge of the convolutional filter.
Comments.UserDisplayName: spiridon_the_sun_rotator
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-04T11:32:03.567Z
Comments.Text: I don't understand why you say that none of the weights is preferred over the other. That's exactly the point of having different weights, even if they are slightly different. Maybe you're confusing this with properties like "translation invariance"?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-04T15:05:34.367Z
Comments.Text: @nbro I understand, that they have to be different. I mean, that the weights $w_{-1}. w_{1}$ would be most likely less important, than $w_{0}$, something like `hamming` or `hann` window decaying behavior at the ends.
Comments.UserDisplayName: spiridon_the_sun_rotator
Tags: <convolutional-neural-networks><reference-request><transformer><convolution><positional-encoding>
Title: Has positional encoding been used in convolutional layers?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # Has positional encoding been used in convolutional layers?
  
  (Asked by: username_0 on 2021-05-31)
  
  Positional encoding (PE) is an essential part of the self-attention layers in the transformer architectures since without adding it in some way (fixed of learnable) to the input embeddings model has ultimately no notion of order and is permutationally equivariant and the given token attends to the far separate and local tokens identically.
  
  The convolution operation with a local filter, say of size $3, 5$ for 1D convolutions or $3 \times 3, 5 \times 5$ for 2D convolutions, has some notion of locality by construction. However, within this neighborhood, all pixels are treated in the same way.
  
  However, it may be the case, that it is important, that the given pixel is the central for the application of this filter, whereas the other is close to the boundary. For small filters
  $3 \times 3$ it is probably not an issue, but for the larger - injection of PE can be useful.
  
  Has this question been investigated in the literature? Are there any architectures with PE + convolutions?
  
  **Tags:** convolutional-neural-networks, reference-request, transformer, convolution, positional-encoding
  > **Comments:**
  > **username_1**: "However, within this neighborhood, all pixels are treated in the same way.", this is not true (unless I'm misinterpreting this statement): that's the point of learning the values of the kernel.  
  > **username_0**: @nbro well, probably my question is really ill-posed, but I mean, that in the 1D conv for example we have a stuff like $w_{-1} x_{i - 1} + w_{0} x_{i} + w_{1} x_{i + 1}$, and here none of the weights and locations $w_{j}, x_{i-j}$ is preferred over the other. However, i would think, that for long enough filters at least, values in the middle are more important, that on the edge of the convolutional filter.  
  > **username_1**: I don't understand why you say that none of the weights is preferred over the other. That's exactly the point of having different weights, even if they are slightly different. Maybe you're confusing this with properties like "translation invariance"?  
  > **username_0**: @nbro I understand, that they have to be different. I mean, that the weights $w_{-1}. w_{1}$ would be most likely less important, than $w_{0}$, something like `hamming` or `hann` window decaying behavior at the ends.  
  
  ---
  
  ## 0 answers
  
  