Answers.Body: <p><strong>The correct interpretation</strong> (based on the comment by the author of the question below):</p>
  <p>Yep, you are right. It is actually only a single cell per object that contributes to the loss with <span class="math-container">$\mathbb{1}^\text{obj}_{ij}$</span> factor. That cell is identified as the one that contains the centre(oid) of the ground truth box of the corresponding object.</p>
  <p><strong>My original (incorrect) interpretation of the paper:</strong></p>
  <p>From what I read in the paper it is not a single bounding box <em>for</em> <em>all</em> cells in the grid for a particular object (your original suggestion).</p>
  <p>Rather for every object and every cell that is assigned to this object only one bounding box contributes to the loss. The network generates <span class="math-container">$B$</span> bounding boxes for every cell in the grid, but we pick only one and only when that cell actually belongs to an object.</p>
  <p>Every cell <span class="math-container">$i$</span>, which is not a background cell, will have exactly one box <span class="math-container">$j$</span>, such that <span class="math-container">$\mathbb{1}^\text{obj}_{ij} = 1$</span></p>
  <p>This is based on the following paragraph from the paper:</p>
  <blockquote>
  <p>Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is &quot;responsible&quot; for the ground truth box <strong>(i.e. has the highest IOU of any predictor in <em>that</em> grid cell).</strong></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-08T15:47:16.817Z
Answers.Text: Thank you for your reply. I've done some digging after I posted this question and found one video where Joseph Redmon(author of the paper) gives a presentation about YOLO: https://youtu.be/NM6lrxy0bxs . Somewhere about 8 min he starts to explain how they train the network and it seems like my initial assumption is rights: 1^{obj}_{ij}=1 only when the center of the ground truth box falls into i cell. Also, it seems like 1^{noobj}_{ij}=1 for every cell where there is no center of ground truth box.
Answers.UserDisplayName: Andrew
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-08T15:58:14.600Z
Answers.Text: yep, looking at the video you are totally right. Still it was nice to clarify how YOLO objective works:)
Answers.UserDisplayName: hellmean
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-03-08T14:42:50.363Z
Answers.LastActivityDate: 2021-01-28T23:12:26.270Z
Answers.LastEditDate: 2021-01-28T23:12:26.270Z
Answers.UserDisplayName: hellmean
Body: <p>I'm trying to implement a custom version of the YOLO neural network. Originally, it was described in the paper <a href="https://arxiv.org/pdf/1506.02640.pdf" rel="nofollow noreferrer">You Only Look Once: Unified, Real-Time Object Detection</a> (2016). I have some problems understanding the loss function they used.</p>
  <p>Basic information:</p>
  <ul>
  <li><p>An input image is divided into <span class="math-container">$S \times S$</span> grid (that gives the total of <span class="math-container">$S^2$</span> cells) and each cell predicts <span class="math-container">$B$</span> bounding boxes and <span class="math-container">$c$</span> conditional class probabilities. Each bounding box predicts <span class="math-container">$5$</span> values: <span class="math-container">$x,y,w,h,C$</span> (center of bounding box, width and height and confidence score). This makes the output of YOLO an <span class="math-container">$S \times S \times (5B*c)$</span> tensor.</p>
  </li>
  <li><p>The <span class="math-container">$(x,y)$</span> coordinates are calculated relative to the bounds of the cell and <span class="math-container">$(w,h)$</span> is relative to the whole image.</p>
  </li>
  <li><p>I understand that the first term penalizes the wrong prediction of the center of a bounding box; the 2nd term penalizes wrong width and height prediction; the 3rd term the wrong confidence prediction; the 4th is responsible for pushing confidence to zero when there is no object in a cell; the last term penalizes wrong class prediction.</p>
  </li>
  </ul>
  <p>My problem:</p>
  <p>I don't understand when <span class="math-container">$\mathbb{1}^\text{obj}_{ij}$</span> should be <span class="math-container">$1$</span> or <span class="math-container">$0$</span>. In the paper, they write (section <strong>2.2. Training</strong>):</p>
  <blockquote>
  <p><span class="math-container">$\mathbb{1}_{i j}^{\mathrm{obj}}$</span> denotes that the <span class="math-container">$j$</span>th bounding box predictor in cell <span class="math-container">$i$</span> is &quot;responsible&quot; for that prediction.</p>
  </blockquote>
  <p>and they also write</p>
  <blockquote>
  <p>Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is &quot;responsible&quot; for the ground truth box</p>
  </blockquote>
  <ul>
  <li><p>So, is it right that, for every object in the image, there should be exactly one pair of <span class="math-container">$ij$</span> such that <span class="math-container">$\mathbb{1}_{i j}^{\mathrm{obj}} = 1$</span>?</p>
  <ul>
  <li><p>If this is correct, this means that the center of the ground truth bounding box should fall into <span class="math-container">$i$</span>th cell, right?</p>
  </li>
  <li><p>If this is not the case, what are other possibilities when <span class="math-container">$\mathbb{1}_{i j}^{\mathrm{obj}} = 1$</span>, and what ground truth labels <span class="math-container">$x_i$</span> and <span class="math-container">$y_i$</span> should be in these cases?</p>
  </li>
  </ul>
  </li>
  <li><p>Also, I assume that ground truth <span class="math-container">$p_i(c)$</span> should be <span class="math-container">$1$</span> if there is an object of class <span class="math-container">$c$</span> in the cell <span class="math-container">$i$</span>, but what ground truth <span class="math-container">$p_i(c)$</span> should be equal to in case there are several objects of different classes in the cell?</p>
  </li>
  </ul>
  <p><a href="https://i.sstatic.net/MPXKM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/MPXKM.png" alt="YOlO loss function" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-28T23:22:13.110Z
Comments.Text: [**Please, next time ask only 1 question per post**](https://meta.stackexchange.com/questions/39223/one-post-with-multiple-questions-or-multiple-posts/39224#39224). If you have multiple questions, even though they are about the same model or topic, split them into multiple posts, so that people can focus on 1 question/problem at a time.
Comments.UserDisplayName: nbro
Tags: <deep-learning><convolutional-neural-networks><papers><object-recognition><yolo>
Title: In YOLO, when is $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what are the ground-truth labels for $x_i$ and $y_i$?
UserDisplayName: Andrew
ThreadText: # In YOLO, when is $\mathbb{1}_{i j}^{\mathrm{obj}} = 1$, and what are the ground-truth labels for $x_i$ and $y_i$?
  
  (Asked by: username_0 on 2018-03-05)
  
  I'm trying to implement a custom version of the YOLO neural network. Originally, it was described in the paper [You Only Look Once: Unified, Real-Time Object Detection](https://arxiv.org/pdf/1506.02640.pdf) (2016). I have some problems understanding the loss function they used.
  
  Basic information:
  
  * An input image is divided into $S \times S$ grid (that gives the total of $S^2$ cells) and each cell predicts $B$ bounding boxes and $c$ conditional class probabilities. Each bounding box predicts $5$ values: $x,y,w,h,C$ (center of bounding box, width and height and confidence score). This makes the output of YOLO an $S \times S \times (5B\*c)$ tensor.
  * The $(x,y)$ coordinates are calculated relative to the bounds of the cell and $(w,h)$ is relative to the whole image.
  * I understand that the first term penalizes the wrong prediction of the center of a bounding box; the 2nd term penalizes wrong width and height prediction; the 3rd term the wrong confidence prediction; the 4th is responsible for pushing confidence to zero when there is no object in a cell; the last term penalizes wrong class prediction.
  
  My problem:
  
  I don't understand when $\mathbb{1}^\text{obj}\_{ij}$ should be $1$ or $0$. In the paper, they write (section **2.2. Training**):
  
  > $\mathbb{1}\_{i j}^{\mathrm{obj}}$ denotes that the $j$th bounding box predictor in cell $i$ is "responsible" for that prediction.
  
  and they also write
  
  > Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box
  
  * So, is it right that, for every object in the image, there should be exactly one pair of $ij$ such that $\mathbb{1}\_{i j}^{\mathrm{obj}} = 1$?
  
    + If this is correct, this means that the center of the ground truth bounding box should fall into $i$th cell, right?
    + If this is not the case, what are other possibilities when $\mathbb{1}\_{i j}^{\mathrm{obj}} = 1$, and what ground truth labels $x\_i$ and $y\_i$ should be in these cases?
  * Also, I assume that ground truth $p\_i(c)$ should be $1$ if there is an object of class $c$ in the cell $i$, but what ground truth $p\_i(c)$ should be equal to in case there are several objects of different classes in the cell?
  
  [![YOlO loss function](https://i.sstatic.net/MPXKM.png)](https://i.sstatic.net/MPXKM.png)
  
  **Tags:** deep-learning, convolutional-neural-networks, papers, object-recognition, yolo
  > **Comments:**
  > **username_1**: [**Please, next time ask only 1 question per post**](https://meta.stackexchange.com/questions/39223/one-post-with-multiple-questions-or-multiple-posts/39224#39224). If you have multiple questions, even though they are about the same model or topic, split them into multiple posts, so that people can focus on 1 question/problem at a time.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-03-08. Score: 1
  
  **The correct interpretation** (based on the comment by the author of the question below):
  
  Yep, you are right. It is actually only a single cell per object that contributes to the loss with $\mathbb{1}^\text{obj}\_{ij}$ factor. That cell is identified as the one that contains the centre(oid) of the ground truth box of the corresponding object.
  
  **My original (incorrect) interpretation of the paper:**
  
  From what I read in the paper it is not a single bounding box *for* *all* cells in the grid for a particular object (your original suggestion).
  
  Rather for every object and every cell that is assigned to this object only one bounding box contributes to the loss. The network generates $B$ bounding boxes for every cell in the grid, but we pick only one and only when that cell actually belongs to an object.
  
  Every cell $i$, which is not a background cell, will have exactly one box $j$, such that $\mathbb{1}^\text{obj}\_{ij} = 1$
  
  This is based on the following paragraph from the paper:
  
  > Note that the loss function only penalizes classification error if an object is present in that grid cell (hence the conditional class probability discussed earlier). It also only penalizes bounding box coordinate error if that predictor is "responsible" for the ground truth box **(i.e. has the highest IOU of any predictor in *that* grid cell).**
  
  > **Comments:**
  > **username_0**: Thank you for your reply. I've done some digging after I posted this question and found one video where Joseph Redmon(author of the paper) gives a presentation about YOLO: https://youtu.be/NM6lrxy0bxs . Somewhere about 8 min he starts to explain how they train the network and it seems like my initial assumption is rights: 1^{obj}_{ij}=1 only when the center of the ground truth box falls into i cell. Also, it seems like 1^{noobj}_{ij}=1 for every cell where there is no center of ground truth box.  
  > **username_2**: yep, looking at the video you are totally right. Still it was nice to clarify how YOLO objective works:)  
  
  ---
  
  