Answers.Body: <h2>TL;TR</h2>
  <p>The location of the input that a neuron sees is called receptive field.</p>
  <p>grid cell is merely used to demonstrate the concept of anchor. It has nothing to do with the receptive field of the network structure.
  The actual receptive field size (RF) depends on the paramters of yolo structure, different size of the yolo network actually has different RF.
  The RF is definitely not the full size of the input image, but it is also way larger than the grid cell. Usually several times bigger.
  The smaller the feature map, the bigger the ratio of RF/cell</p>
  <p>For yolov8 tiny, the RF of P3 is 12 times of P3 grid cell size. P4 is 7x, P5 is 4x.</p>
  <p>yolo use each P3-P5 to predict exactly one bonding box per neuron.
  For P3 it is 80x80 neurons, for P4 it is 40x40 neurons, for P5 it is 20x20 neurons.</p>
  <p>The <strong>size</strong> of the object can be predict for each neuron is <strong>not constrained</strong> at all. It only limit the location of the object. see detail section for more discussion.</p>
  <p>During training, one grid cell only provide a <strong>single</strong> box, but same ground truth box can be assigned to <strong>multiple</strong> grid cells.
  Thus during inference it is possible to have multiple grid cells per feature map predict a single instance.</p>
  <p>To handle this multiple-to-one problem, we need a postprocess step to select one box for each instance. During inference, the model predict additional score: class-quality score which roughly means the class probability multiple by box quality. This is then used as likely-hood score or confidence for corresponding box:</p>
  <ol>
  <li>we first use a threshold of the confidence to filter out the low quality bbox.</li>
  <li>for each bbox it choose the argmax of class confidence as the instance class.</li>
  <li>If we are lucky there is no further postprocess is required. However, it is very common, that we still have some redundant boxes per instance. There are boxes per each class have large overlaps. To reduce the redundancy we use a heuristic algorithm called NMS. It is heuristic, because in high dense scenario it is possible to have multiple instance closely overlapped, but with NMS we can only detect one of these instances.</li>
  <li>the NMS basically applied to each class separately. It basically start from high confidence box then remove the boxes has lower confidence but high overlap ratio</li>
  </ol>
  <h2>some details about the relationship of grid cell, predict box and receptive field.</h2>
  <p>A neuron can only predict a box contains the grid cell centre, but the box size can be much larger or much smaller.
  The extreme case is demonstrated in following image.
  The predicted box can be as big as the image itself.
  There is no limit of the predicted box size relative to either grid cell or receptive field.</p>
  <p><a href="https://i.sstatic.net/bZABbjWU.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bZABbjWU.jpg" alt="enter image description here" /></a></p>
  <p>So why we allow it to predict a box larger than the input (receptive field)?
  Because the network can &quot;guess&quot; the actual box via seeing partial of the object. For example, even though we can only see part of the bicycle, we can still estimate how big the actual bicycle is:</p>
  <p><a href="https://i.sstatic.net/YjozBwJx.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YjozBwJx.jpg" alt="enter image description here" /></a></p>
  <h2>more details about the receptive field</h2>
  <p>Let's take the modern yolov8 as example. To simplify stuff we only look at the backbone.
  It is a CNN structure.</p>
  <p>In CNN the receptive field only changed when the conv kernel size &gt; 1 or stride size &gt; 1:</p>
  <ul>
  <li>The conv has k=1, s=1, is only a channel mixer which does not blend any lateral spatial info.</li>
  <li>when k&gt;1 each neuron see kxk input</li>
  <li>when s&gt;1, the current layer skip some input while scanning. Thus equivalently it makes the next layer's neuron seeing wider input.</li>
  </ul>
  <p>Thus the RF can be calculate as:
  <span class="math-container">$$
  RF_{l} = RF_{l-1} + (K_{l} - 1) \times \prod_{j=1}^{l-1} S_{j}
  $$</span></p>
  <p>Where:</p>
  <ul>
  <li><span class="math-container">$RF_{l}$</span> is the receptive field of layer <span class="math-container">$l$</span>.</li>
  <li><span class="math-container">$K_{l}$</span> is the kernel size of layer <span class="math-container">$l$</span>.</li>
  <li><span class="math-container">$S_{j}$</span> is the stride of layer <span class="math-container">$j$</span>.</li>
  </ul>
  <p><a href="https://github.com/ultralytics/ultralytics/issues/189" rel="nofollow noreferrer">Here</a> is the actual structure overview. Focus on the left side backend part.</p>
  <p>the C2f is tricky, it contains n bottlnecks each of the bottleneck has 4 conv modules. For tiny model the d = 0.33</p>
  <p>We can roughly see it as</p>
  <ul>
  <li>stem conv</li>
  <li>C1 conv</li>
  <li>C2 C2f n=1</li>
  <li>conv</li>
  <li>C3 C2f n=2</li>
  <li>conv</li>
  <li>C4 C2f n=2</li>
  <li>conv</li>
  <li>C2f C2f n=1</li>
  <li>C5 (SPPF) (9)</li>
  </ul>
  <p>So the kernel size and stride for each layer can be represent as a list of list: blocks</p>
  <pre><code># each bottleneck blcok has 4 conv that k&gt;1
  btn = [(3,1)]*4
  # stem, C1 (1), C2 (2), conv, C3 (4), conv, C4 (6), conv, C2f, C5 (SPPF) (9)
  idx_Cs = [1,2,4,6,9]
  blocks = [
    [(3,2)], # conv
    [(3,2)], # conv
    btn*1, # C2f n=3*d
    [(3,2)], # conv
    btn*2, # C2f n=6*d
    [(3,2)],
    btn*2, # C2f n=6*d
    [(3,2)],
    btn*1, # C2f n=3*d
    [(3,1), (5,1), (5,1), (5,1), (3, 1)] # SPPF
  ]
  </code></pre>
  <p>For the backbone we already have recptive field for each feature map like following:</p>
  <pre><code>C1 has RF= 7x7
  C2 has RF= 15x15
  C3 has RF= 35x35
  C4 has RF= 55x55
  C5 has RF= 83x83
  </code></pre>
  <p>However, we still have FPN which continues to fuse the feature map to form P3, P4, P5.
  This further increases the receptive field, because for each Pn, the longest path is longer than C5.
  I will not put details here but I can show the final results:</p>
  <pre><code>P3 has RF= 99x99
  P4 has RF= 111x111
  P5 has RF= 123x123
  </code></pre>
  <p>detection heads then add another 2 convs that can change RF. Resulting the final RF:</p>
  <pre><code>DET_P3 has RF= 103x103
  DET_P4 has RF= 115x115
  DET_P5 has RF= 127x127
  </code></pre>
  <p>Remember, for P3, P4, and P5, the reducing factor is 8, 16, 32.
  This means the grid cell size in pixel is: 8x8, 16x16 and 32x32.
  We can see they are far smaller than the actual receptive fields.</p>
  <p>Now you also understand the limit of the yolo structure, it usually struggling to predict big object which cover most of the image, since the RF of P5 does not cover the entire input.
  This is also why there is enhanced version that has P6 layer to add a bigger feature map.</p>
  <p>You can find the code used to compute the RF <a href="https://gist.github.com/braindevices/7a7d290acfb32a6763d51f25f73ec302" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-06-30T00:38:15.843Z
Answers.LastActivityDate: 2024-06-30T13:23:03.750Z
Answers.LastEditDate: 2024-06-30T13:23:03.750Z
Answers.UserDisplayName: Wang
Answers.Body: <p>Lets say you split the image by 13x13 (SxS) cells. For each grid cell, you predict 5 (older YOLO) or 9 (newer versions) bounding boxes.</p>
  <p>These (<em>anchors</em>) are supposed to have various shapes and scales, so they vertical, horizontal and squared rectangle at small scale, and middle scale and bigger scale.</p>
  <p>So for each cell (13x13) and for each anchor (9) , you predict (width offset ,height offset,center x,center y, objectness, class1 ,class2,... class k ). You actually predict <em>scaling</em> of the width and height of the box, so you can end up with variable sizes.</p>
  <p>Ojectness marks if there is any object in this cell for this anchor, if not, it is discarded later.</p>
  <p>Since you end up with lot of predicted bounding boxes of various shapes, you need to filter them out, this is called <a href="https://learnopencv.com/non-maximum-suppression-theory-and-implementation-in-pytorch/" rel="nofollow noreferrer">non maximum supression</a>.
  For each anchor you take class with highest confidence.</p>
  <p><strong>EDIT:</strong></p>
  <p>So to answer your questions, If an object is in multiple cells, it is contained in multiple predicted boxes, and you take the one with maximum confidence and discard others.</p>
  <p>Each cell does not see <em>only part</em> of the image, it sees the <strong>whole</strong> image. Before you get to 13x13 cells, you have 50 or more layers of <strong>conv &amp; max pooling</strong> layers, which gives each cell <strong>receptive field</strong> of the whole image.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-24T13:18:22.830Z
Answers.Text: Firstly, thank you very much for taking your valuable time to answer my question. Secondly, can you please brief me a little more about anchors boxes? Are bounding boxes and anchors boxes mean the same thing? Are their positions are fixed for each grid? What if the objects that we are looking for doesn't falls in any of the 9 anchors boxes? What do we do then? Do we get any prediction if objects doesn't falls in those 9 anchors boxes or we don't?
Answers.UserDisplayName: Sharjeel M. Rajput
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-20T17:40:37.440Z
Answers.LastActivityDate: 2022-06-20T17:47:54.890Z
Answers.LastEditDate: 2022-06-20T17:47:54.890Z
Answers.UserDisplayName: Martin
Body: <p>I have been reading various articles and watching videos on YouTube, but I can't seem to understand one thing.</p>
  <p>How does YOLO make a bounding box for an object if it is in multiple grid cells? For example, in the picture given below, how does it predicts the bounding box for the classes, because they fall in multiple cells? How does it know what object is in a grid cell even when it sees a small part of it?</p>
  <p>It's been very difficult for me to get these answers.</p>
  <p><a href="https://i.sstatic.net/NvY8e.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NvY8e.jpg" alt="" /></a></p>
  
Tags: <deep-learning><convolutional-neural-networks><object-detection><yolo><bounding-box>
Title: How does YOLO detect the object when the object is in multiple grid cells?
UserDisplayName: Sharjeel M. Rajput
ThreadText: # How does YOLO detect the object when the object is in multiple grid cells?
  
  (Asked by: username_0 on 2022-06-15)
  
  I have been reading various articles and watching videos on YouTube, but I can't seem to understand one thing.
  
  How does YOLO make a bounding box for an object if it is in multiple grid cells? For example, in the picture given below, how does it predicts the bounding box for the classes, because they fall in multiple cells? How does it know what object is in a grid cell even when it sees a small part of it?
  
  It's been very difficult for me to get these answers.
  
  [![](https://i.sstatic.net/NvY8e.jpg)](https://i.sstatic.net/NvY8e.jpg)
  
  **Tags:** deep-learning, convolutional-neural-networks, object-detection, yolo, bounding-box
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2024-06-30. Score: 0
  
  ## TL;TR
  
  The location of the input that a neuron sees is called receptive field.
  
  grid cell is merely used to demonstrate the concept of anchor. It has nothing to do with the receptive field of the network structure.
  The actual receptive field size (RF) depends on the paramters of yolo structure, different size of the yolo network actually has different RF.
  The RF is definitely not the full size of the input image, but it is also way larger than the grid cell. Usually several times bigger.
  The smaller the feature map, the bigger the ratio of RF/cell
  
  For yolov8 tiny, the RF of P3 is 12 times of P3 grid cell size. P4 is 7x, P5 is 4x.
  
  yolo use each P3-P5 to predict exactly one bonding box per neuron.
  For P3 it is 80x80 neurons, for P4 it is 40x40 neurons, for P5 it is 20x20 neurons.
  
  The **size** of the object can be predict for each neuron is **not constrained** at all. It only limit the location of the object. see detail section for more discussion.
  
  During training, one grid cell only provide a **single** box, but same ground truth box can be assigned to **multiple** grid cells.
  Thus during inference it is possible to have multiple grid cells per feature map predict a single instance.
  
  To handle this multiple-to-one problem, we need a postprocess step to select one box for each instance. During inference, the model predict additional score: class-quality score which roughly means the class probability multiple by box quality. This is then used as likely-hood score or confidence for corresponding box:
  
  1. we first use a threshold of the confidence to filter out the low quality bbox.
  2. for each bbox it choose the argmax of class confidence as the instance class.
  3. If we are lucky there is no further postprocess is required. However, it is very common, that we still have some redundant boxes per instance. There are boxes per each class have large overlaps. To reduce the redundancy we use a heuristic algorithm called NMS. It is heuristic, because in high dense scenario it is possible to have multiple instance closely overlapped, but with NMS we can only detect one of these instances.
  4. the NMS basically applied to each class separately. It basically start from high confidence box then remove the boxes has lower confidence but high overlap ratio
  
  ## some details about the relationship of grid cell, predict box and receptive field.
  
  A neuron can only predict a box contains the grid cell centre, but the box size can be much larger or much smaller.
  The extreme case is demonstrated in following image.
  The predicted box can be as big as the image itself.
  There is no limit of the predicted box size relative to either grid cell or receptive field.
  
  [![enter image description here](https://i.sstatic.net/bZABbjWU.jpg)](https://i.sstatic.net/bZABbjWU.jpg)
  
  So why we allow it to predict a box larger than the input (receptive field)?
  Because the network can "guess" the actual box via seeing partial of the object. For example, even though we can only see part of the bicycle, we can still estimate how big the actual bicycle is:
  
  [![enter image description here](https://i.sstatic.net/YjozBwJx.jpg)](https://i.sstatic.net/YjozBwJx.jpg)
  
  ## more details about the receptive field
  
  Let's take the modern yolov8 as example. To simplify stuff we only look at the backbone.
  It is a CNN structure.
  
  In CNN the receptive field only changed when the conv kernel size > 1 or stride size > 1:
  
  * The conv has k=1, s=1, is only a channel mixer which does not blend any lateral spatial info.
  * when k>1 each neuron see kxk input
  * when s>1, the current layer skip some input while scanning. Thus equivalently it makes the next layer's neuron seeing wider input.
  
  Thus the RF can be calculate as:
  $$
  RF\_{l} = RF\_{l-1} + (K\_{l} - 1) \times \prod\_{j=1}^{l-1} S\_{j}
  $$
  
  Where:
  
  * $RF\_{l}$ is the receptive field of layer $l$.
  * $K\_{l}$ is the kernel size of layer $l$.
  * $S\_{j}$ is the stride of layer $j$.
  
  [Here](https://github.com/ultralytics/ultralytics/issues/189) is the actual structure overview. Focus on the left side backend part.
  
  the C2f is tricky, it contains n bottlnecks each of the bottleneck has 4 conv modules. For tiny model the d = 0.33
  
  We can roughly see it as
  
  * stem conv
  * C1 conv
  * C2 C2f n=1
  * conv
  * C3 C2f n=2
  * conv
  * C4 C2f n=2
  * conv
  * C2f C2f n=1
  * C5 (SPPF) (9)
  
  So the kernel size and stride for each layer can be represent as a list of list: blocks
  
  ```
  # each bottleneck blcok has 4 conv that k>1
  btn = [(3,1)]*4
  # stem, C1 (1), C2 (2), conv, C3 (4), conv, C4 (6), conv, C2f, C5 (SPPF) (9)
  idx_Cs = [1,2,4,6,9]
  blocks = [
    [(3,2)], # conv
    [(3,2)], # conv
    btn*1, # C2f n=3*d
    [(3,2)], # conv
    btn*2, # C2f n=6*d
    [(3,2)],
    btn*2, # C2f n=6*d
    [(3,2)],
    btn*1, # C2f n=3*d
    [(3,1), (5,1), (5,1), (5,1), (3, 1)] # SPPF
  ]
  
  ```
  
  For the backbone we already have recptive field for each feature map like following:
  
  ```
  C1 has RF= 7x7
  C2 has RF= 15x15
  C3 has RF= 35x35
  C4 has RF= 55x55
  C5 has RF= 83x83
  
  ```
  
  However, we still have FPN which continues to fuse the feature map to form P3, P4, P5.
  This further increases the receptive field, because for each Pn, the longest path is longer than C5.
  I will not put details here but I can show the final results:
  
  ```
  P3 has RF= 99x99
  P4 has RF= 111x111
  P5 has RF= 123x123
  
  ```
  
  detection heads then add another 2 convs that can change RF. Resulting the final RF:
  
  ```
  DET_P3 has RF= 103x103
  DET_P4 has RF= 115x115
  DET_P5 has RF= 127x127
  
  ```
  
  Remember, for P3, P4, and P5, the reducing factor is 8, 16, 32.
  This means the grid cell size in pixel is: 8x8, 16x16 and 32x32.
  We can see they are far smaller than the actual receptive fields.
  
  Now you also understand the limit of the yolo structure, it usually struggling to predict big object which cover most of the image, since the RF of P5 does not cover the entire input.
  This is also why there is enhanced version that has P6 layer to add a bigger feature map.
  
  You can find the code used to compute the RF [here](https://gist.github.com/braindevices/7a7d290acfb32a6763d51f25f73ec302).
  
  
  ---
  
  ### Answer by username_2 on 2022-06-20. Score: 0
  
  Lets say you split the image by 13x13 (SxS) cells. For each grid cell, you predict 5 (older YOLO) or 9 (newer versions) bounding boxes.
  
  These (*anchors*) are supposed to have various shapes and scales, so they vertical, horizontal and squared rectangle at small scale, and middle scale and bigger scale.
  
  So for each cell (13x13) and for each anchor (9) , you predict (width offset ,height offset,center x,center y, objectness, class1 ,class2,... class k ). You actually predict *scaling* of the width and height of the box, so you can end up with variable sizes.
  
  Ojectness marks if there is any object in this cell for this anchor, if not, it is discarded later.
  
  Since you end up with lot of predicted bounding boxes of various shapes, you need to filter them out, this is called [non maximum supression](https://learnopencv.com/non-maximum-suppression-theory-and-implementation-in-pytorch/).
  For each anchor you take class with highest confidence.
  
  **EDIT:**
  
  So to answer your questions, If an object is in multiple cells, it is contained in multiple predicted boxes, and you take the one with maximum confidence and discard others.
  
  Each cell does not see *only part* of the image, it sees the **whole** image. Before you get to 13x13 cells, you have 50 or more layers of **conv & max pooling** layers, which gives each cell **receptive field** of the whole image.
  
  > **Comments:**
  > **username_0**: Firstly, thank you very much for taking your valuable time to answer my question. Secondly, can you please brief me a little more about anchors boxes? Are bounding boxes and anchors boxes mean the same thing? Are their positions are fixed for each grid? What if the objects that we are looking for doesn't falls in any of the 9 anchors boxes? What do we do then? Do we get any prediction if objects doesn't falls in those 9 anchors boxes or we don't?  
  
  ---
  
  