Answers.Body: <p>A filter value in a smaller deeper layer is in a fixed relationship to the geometry of the input.</p>
  <p>Its position in the filter defines its relation to an image.</p>
  <p>But positional encoding is actually needed and common in transformers etc, where that is not the case.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-15T20:53:50.253Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-16T07:59:10.787Z
Answers.Text: Well, I see what you say that the deeper you go, the bigger is the field of view and the features extracted will be related to that field of view and nothing else. However, you still have the "problem" of not knowing to what exact region of the image (coordinates x,y) does those features belong.
Answers.UserDisplayName: IgnacioGaBo
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-15T15:14:34.870Z
Answers.LastActivityDate: 2023-03-15T15:14:34.870Z
Answers.UserDisplayName: Volker Siegel
Answers.Body: <p>In a purely convolutional network, shift equivariance is inherent (ignoring boundary effects), meaning the network lacks an awareness of specific x/y coordinates. Introducing positional information by overlaying the pixel grid with coordinates would enable the model to gain a sense of distance within the image.</p>
  <p>However, incorporating absolute positional encoding would break this shift equivariance. Consequently, the model would lose this advantageous property and would function more similarly to applying an MLP directly to the entire image.</p>
  <p>It is important to note that shift equivariance is typically considered a beneficial feature of CNNs, as it allows for the detection of objects regardless of their location within the image.</p>
  <p>Nonetheless, if you require a sense of absolute position while still retaining the shared parameter architecture of CNNs, you can indeed opt for absolute positional encoding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-08-07T14:52:05.560Z
Answers.LastActivityDate: 2024-08-07T15:11:54.373Z
Answers.LastEditDate: 2024-08-07T15:11:54.373Z
Answers.UserDisplayName: ju2ez
Body: <p>A CNN model is based on a series of filters applied to an image. However, these filters can only &quot;see&quot; a small portion of the image and they have no information of the relative position of the pixels that they are analyzing with respect to the image coordinates. Likewise, deeper in the network, they have no information of the the position of the field of view that they are analyzing with respect to the coordinate-system of the image. In other words, if <a href="https://arxiv.org/abs/2110.05861" rel="nofollow noreferrer">trained correctly, these modes are translation-invariant</a> and they do not &quot;know&quot; the region of the image to which some features belong (ie, they have no information if some extracted features belong to the lower, upper, right or left part of the image).</p>
  <p>I was wondering how a model like, for example, <a href="https://arxiv.org/abs/1506.02640" rel="nofollow noreferrer">YOLO</a> can infer the relative coordinates of the objects that it is detecting, without having this information. I understand that the image is divided in in a grid of cells and that gives some notion of region (ie, part of the image that is being analyzed). However, we still have to guess the width, height and relative x,y position with respect to the cell. Why don't we overlap the input image (or extracted features) with some kind of positional encoding (the x,y coordinates for example), so that the model has information of the region of the image to which some features belong?</p>
  <p>This question also applies to other CNN models like, for example, U-Net architectures. I think that it may be useful to have some information of the position of the pixels/feature map with respect to the image's coordinates in some problems. For example, to <a href="https://ieeexplore.ieee.org/document/9121887" rel="nofollow noreferrer">segment the road</a> in self-driving problems, it will be very useful to &quot;know&quot; if the region of the image that is being analyzed is in the bottom part of the image as, most-likely, the road will be found there.</p>
  <p>How does these models infer this information? And, why is not common to add this &quot;positional encoding&quot; in deep learning projects applied to images?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-15T21:06:19.237Z
Comments.Text: Can you explain what you mean when you say it infers the coordinates? do you give it a single pixel and it tells you x=100 y=200?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-16T08:52:47.067Z
Comments.Text: With this I mean that it is some useful information. It depends on the model, but in some problems, knowing that some pixels belong to a particular region of the image may be useful (for example, when segmenting the road it is usefull to know if the region is the lower part of the image or the upper part). As well, some models like YOLO directly outputs the coordinates (x, y, w, h) of the objects it is detecting.
Comments.UserDisplayName: IgnacioGaBo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-16T12:08:04.280Z
Comments.Text: After the convolutional layers there are usually dense layers.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-16T15:47:35.200Z
Comments.Text: You are right. I guess that the dense layers may give a sense of spatial location as they are analyzing the image as a whole. However, I see it as a very indirect way to get this sense of spatial location. Why don't we usually add a positional encoding to the input directly? In addition, you don't have this in purely convolutional models such as CNNs for segmentation.
Comments.UserDisplayName: IgnacioGaBo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-16T15:54:40.873Z
Comments.Text: I don't know. You could try it and see. For segmentation the network is probably *not* aware of which part of the picture it's looking at although I suppose some positional encoding could help it decide (e.g. sky is at the top)
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <deep-learning><convolutional-neural-networks><object-detection><deep-neural-networks>
Title: How vision models based on CNNs learn the relative positions of each pixel in the image?
UserDisplayName: IgnacioGaBo
ThreadText: # How vision models based on CNNs learn the relative positions of each pixel in the image?
  
  (Asked by: username_0 on 2023-03-15)
  
  A CNN model is based on a series of filters applied to an image. However, these filters can only "see" a small portion of the image and they have no information of the relative position of the pixels that they are analyzing with respect to the image coordinates. Likewise, deeper in the network, they have no information of the the position of the field of view that they are analyzing with respect to the coordinate-system of the image. In other words, if [trained correctly, these modes are translation-invariant](https://arxiv.org/abs/2110.05861) and they do not "know" the region of the image to which some features belong (ie, they have no information if some extracted features belong to the lower, upper, right or left part of the image).
  
  I was wondering how a model like, for example, [YOLO](https://arxiv.org/abs/1506.02640) can infer the relative coordinates of the objects that it is detecting, without having this information. I understand that the image is divided in in a grid of cells and that gives some notion of region (ie, part of the image that is being analyzed). However, we still have to guess the width, height and relative x,y position with respect to the cell. Why don't we overlap the input image (or extracted features) with some kind of positional encoding (the x,y coordinates for example), so that the model has information of the region of the image to which some features belong?
  
  This question also applies to other CNN models like, for example, U-Net architectures. I think that it may be useful to have some information of the position of the pixels/feature map with respect to the image's coordinates in some problems. For example, to [segment the road](https://ieeexplore.ieee.org/document/9121887) in self-driving problems, it will be very useful to "know" if the region of the image that is being analyzed is in the bottom part of the image as, most-likely, the road will be found there.
  
  How does these models infer this information? And, why is not common to add this "positional encoding" in deep learning projects applied to images?
  
  **Tags:** deep-learning, convolutional-neural-networks, object-detection, deep-neural-networks
  
  > **Comments:**
  > **username_1**: Can you explain what you mean when you say it infers the coordinates? do you give it a single pixel and it tells you x=100 y=200?  
  > **username_0**: With this I mean that it is some useful information. It depends on the model, but in some problems, knowing that some pixels belong to a particular region of the image may be useful (for example, when segmenting the road it is usefull to know if the region is the lower part of the image or the upper part). As well, some models like YOLO directly outputs the coordinates (x, y, w, h) of the objects it is detecting.  
  > **username_1**: After the convolutional layers there are usually dense layers.  
  > **username_0**: You are right. I guess that the dense layers may give a sense of spatial location as they are analyzing the image as a whole. However, I see it as a very indirect way to get this sense of spatial location. Why don't we usually add a positional encoding to the input directly? In addition, you don't have this in purely convolutional models such as CNNs for segmentation.  
  > **username_1**: I don't know. You could try it and see. For segmentation the network is probably *not* aware of which part of the picture it's looking at although I suppose some positional encoding could help it decide (e.g. sky is at the top)  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-03-15. Score: 0
  
  A filter value in a smaller deeper layer is in a fixed relationship to the geometry of the input.
  
  Its position in the filter defines its relation to an image.
  
  But positional encoding is actually needed and common in transformers etc, where that is not the case.
  
  > **Comments:**
  > **username_3**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  > **username_0**: Well, I see what you say that the deeper you go, the bigger is the field of view and the features extracted will be related to that field of view and nothing else. However, you still have the "problem" of not knowing to what exact region of the image (coordinates x,y) does those features belong.  
  
  ---
  
  ### Answer by username_4 on 2024-08-07. Score: 0
  
  In a purely convolutional network, shift equivariance is inherent (ignoring boundary effects), meaning the network lacks an awareness of specific x/y coordinates. Introducing positional information by overlaying the pixel grid with coordinates would enable the model to gain a sense of distance within the image.
  
  However, incorporating absolute positional encoding would break this shift equivariance. Consequently, the model would lose this advantageous property and would function more similarly to applying an MLP directly to the entire image.
  
  It is important to note that shift equivariance is typically considered a beneficial feature of CNNs, as it allows for the detection of objects regardless of their location within the image.
  
  Nonetheless, if you require a sense of absolute position while still retaining the shared parameter architecture of CNNs, you can indeed opt for absolute positional encoding.
  
  
  ---
  
  