Answers.Body: <p>If I understood well you have 2 questions.</p>
  <ul>
  <li>How to get the bounding box given the network output</li>
  <li>What Smooth L1 loss is</li>
  </ul>
  <p>The answer to your first question lies in the equation (2) in the section 3.2.1 from the <a href="https://arxiv.org/pdf/1506.01497.pdf" rel="nofollow noreferrer">Faster R-CNN paper</a>. As all anchor based object detector (Faster RCNN, YOLOv3, EfficientNets, FPN...) the regression output from the network are not the bounding box coordinates. The regression output predicts the shift of the predicted bounding box with respect to the selected anchor (all of these networks, use more than 1 anchor per location, check section 3.1.1 from the paper).</p>
  <p>So basically what your network predict is <span class="math-container">$t_x, t_y, t_w, t_h$</span>:</p>
  <p><a href="https://i.sstatic.net/hf2VM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/hf2VM.png" alt="enter image description here" /></a></p>
  <p>And the bounding box coordinates are <span class="math-container">$x, y, w, h$</span>, and the anchor coordinates are <span class="math-container">$x_a, y_a, w_a, h_a$</span>. So in order to compute <span class="math-container">$x, y, w, h$</span> from <span class="math-container">$t_x, t_y, t_w, t_h$</span>, you just have to invert the equations above. However I think you could gain more intuition about it if you take your time and read the whole section 3.1 from the paper. I know sometimes is a pain, but you will grasp the high level concept.</p>
  <p>With regard to your second question. Yes the loss is computed with the output from the network and the &quot;coded&quot; ground truth, meaning you compute the loss with the paramters <span class="math-container">$t$</span> (predicted) against <span class="math-container">$t^*$</span> (coded ground truth) instead for computing loss with the real coordinates of the bounding boxes (decoded output from the network). For the equation on Smooth L1 loss check <a href="https://pytorch.org/docs/master/generated/torch.nn.SmoothL1Loss.html#torch.nn.SmoothL1Loss" rel="nofollow noreferrer">this wonderful documentation</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-03T10:48:41.473Z
Answers.LastActivityDate: 2020-08-03T10:53:47.730Z
Answers.LastEditDate: 2020-08-03T10:53:47.730Z
Answers.UserDisplayName: JVGD
Body: <p>The <a href="https://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks.pdf" rel="nofollow noreferrer">RPN</a> loss in <a href="https://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks.pdf" rel="nofollow noreferrer">Faster RCNN</a> paper is</p>
  <p><span class="math-container">$$
  L({p_i}, {t_i}) = \frac{1}{N_{cls}} \sum_{i} L_{cls}(p_i,p_i^*) + \lambda \frac{1}{N_{reg}} \sum_i p_i^* L_{reg}(t_i, t_i^*)
  $$</span></p>
  <p>For regression problems, we have the following parametrization</p>
  <p><span class="math-container">$$t_x=\frac{x - x_a}{w_a}, \\ t_y=\frac{y−y_a}{h_a}, \\ t_w= \log \left( \frac{w}{w_a} \right),\\ t_h= \log \left(\frac{h}{h_a} \right)$$</span></p>
  <p>and the ground-truth labels are</p>
  <p><span class="math-container">$$t_x^*=\frac{x^* - x_a}{w_a},\\ t_y^*=\frac{y^*−y_a}{h_a}, \\ t_w^*= \log \left( \frac{w^*}{w_a} \right), \\ t_h^*= \log \left(\frac{h^*}{h_a} \right)$$</span></p>
  <p>where</p>
  <ul>
  <li><p><span class="math-container">$x$</span> and <span class="math-container">$y$</span> are the two coordinates of the center, <span class="math-container">$w$</span> the width, and <span class="math-container">$h$</span> the height of the <em>predicted box</em>.</p>
  </li>
  <li><p><span class="math-container">$x$</span> and <span class="math-container">$y$</span> are the two coordinates of the center, <span class="math-container">$w$</span> the width, and <span class="math-container">$h$</span> the height of the <em>anchor box</em>.</p>
  </li>
  <li><p><span class="math-container">$L_{reg}(t_i, t_i^*) = R(t_i − t_i^*)$</span>, where <span class="math-container">$R$</span> is a robust loss function (smooth <span class="math-container">$L_1$</span>)</p>
  </li>
  </ul>
  <p>These equations are unclear to me, so here are my two questions.</p>
  <ol>
  <li><p>How can I get the predicted bounding box given the neural network's output?</p>
  </li>
  <li><p>What exactly is the smooth <span class="math-container">$L_1$</span> here? How is it defined?</p>
  </li>
  </ol>
  
Tags: <convolutional-neural-networks><papers><object-detection><r-cnn><faster-r-cnn>
Title: In Faster R-CNN, how can I get the predicted bounding box given the neural network's output?
UserDisplayName: user31844
ThreadText: # In Faster R-CNN, how can I get the predicted bounding box given the neural network's output?
  
  (Asked by: username_0 on 2019-12-04)
  
  The [RPN](https://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks.pdf) loss in [Faster RCNN](https://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks.pdf) paper is
  
  $$
  L({p\_i}, {t\_i}) = \frac{1}{N\_{cls}} \sum\_{i} L\_{cls}(p\_i,p\_i^\*) + \lambda \frac{1}{N\_{reg}} \sum\_i p\_i^\* L\_{reg}(t\_i, t\_i^\*)
  $$
  
  For regression problems, we have the following parametrization
  
  $$t\_x=\frac{x - x\_a}{w\_a}, \\ t\_y=\frac{y−y\_a}{h\_a}, \\ t\_w= \log \left( \frac{w}{w\_a} \right),\\ t\_h= \log \left(\frac{h}{h\_a} \right)$$
  
  and the ground-truth labels are
  
  $$t\_x^\*=\frac{x^\* - x\_a}{w\_a},\\ t\_y^\*=\frac{y^\*−y\_a}{h\_a}, \\ t\_w^\*= \log \left( \frac{w^\*}{w\_a} \right), \\ t\_h^\*= \log \left(\frac{h^\*}{h\_a} \right)$$
  
  where
  
  * $x$ and $y$ are the two coordinates of the center, $w$ the width, and $h$ the height of the *predicted box*.
  * $x$ and $y$ are the two coordinates of the center, $w$ the width, and $h$ the height of the *anchor box*.
  * $L\_{reg}(t\_i, t\_i^\*) = R(t\_i − t\_i^\*)$, where $R$ is a robust loss function (smooth $L\_1$)
  
  These equations are unclear to me, so here are my two questions.
  
  1. How can I get the predicted bounding box given the neural network's output?
  2. What exactly is the smooth $L\_1$ here? How is it defined?
  
  **Tags:** convolutional-neural-networks, papers, object-detection, r-cnn, faster-r-cnn
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-03. Score: 0
  
  If I understood well you have 2 questions.
  
  * How to get the bounding box given the network output
  * What Smooth L1 loss is
  
  The answer to your first question lies in the equation (2) in the section 3.2.1 from the [Faster R-CNN paper](https://arxiv.org/pdf/1506.01497.pdf). As all anchor based object detector (Faster RCNN, YOLOv3, EfficientNets, FPN...) the regression output from the network are not the bounding box coordinates. The regression output predicts the shift of the predicted bounding box with respect to the selected anchor (all of these networks, use more than 1 anchor per location, check section 3.1.1 from the paper).
  
  So basically what your network predict is $t\_x, t\_y, t\_w, t\_h$:
  
  [![enter image description here](https://i.sstatic.net/hf2VM.png)](https://i.sstatic.net/hf2VM.png)
  
  And the bounding box coordinates are $x, y, w, h$, and the anchor coordinates are $x\_a, y\_a, w\_a, h\_a$. So in order to compute $x, y, w, h$ from $t\_x, t\_y, t\_w, t\_h$, you just have to invert the equations above. However I think you could gain more intuition about it if you take your time and read the whole section 3.1 from the paper. I know sometimes is a pain, but you will grasp the high level concept.
  
  With regard to your second question. Yes the loss is computed with the output from the network and the "coded" ground truth, meaning you compute the loss with the paramters $t$ (predicted) against $t^\*$ (coded ground truth) instead for computing loss with the real coordinates of the bounding boxes (decoded output from the network). For the equation on Smooth L1 loss check [this wonderful documentation](https://pytorch.org/docs/master/generated/torch.nn.SmoothL1Loss.html#torch.nn.SmoothL1Loss).
  
  
  ---
  
  