Answers.Body: <p><a href="https://openaccess.thecvf.com/content_CVPR_2019/papers/Kirillov_Panoptic_Segmentation_CVPR_2019_paper.pdf" rel="nofollow noreferrer">Panoptic Segmentation</a> (PS) is a computer vision (CV) task that aims at <em>unifying</em> both semantic and instance segmentation.</p>
  <p><a href="https://i.sstatic.net/yHead.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yHead.jpg" alt="enter image description here" /></a></p>
  <ul>
  <li><strong>Semantic segmentation:</strong> each pixel is labeled with a category (<span class="math-container">$C$</span> in total), obtaining a <em>dense prediction</em> over an image. The output image has the same resolution of the original one, but only contains the predicted categories of each pixel. If we consider a specific category <span class="math-container">$c$</span> in the output image, we obtain the <em>segmentation mask</em> that ideally corresponds to the area that all the objects belonging to category <span class="math-container">$c$</span> occupy. Usually, the categories <span class="math-container">$C$</span> depict <em>stuff</em> objects: amorphous regions of same texture and/or material, like the roads and walls. Popular dataset are the PASCAL-VOC2, COCO, ADE20k, and CityScapes. Segmentation models are evaluated with a <em>generalized IoU</em> measure over segmentation masks. Differently from object detection, since there is no overlap among pixels there is no need to apply non-maximum suppression to reject multiple predictions. Common neural network architectures are the <a href="https://openaccess.thecvf.com/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf" rel="nofollow noreferrer">FCNs</a>, <a href="https://arxiv.org/pdf/1505.04597.pdf" rel="nofollow noreferrer">U-Net</a>, and <a href="https://arxiv.org/pdf/1606.00915" rel="nofollow noreferrer">DeepLab</a> based ones.</li>
  <li><strong>Instance segmentation:</strong> is about separating different categories at the pixel level into different instances of the same class. Compared to object detection, instance segmentation provides an accurate segmentation of the object (instead of a rectangular region) that localizes it in the space. Each detected instance of the same category is assigned an ID. <a href="http://openaccess.thecvf.com/content_ICCV_2017/papers/He_Mask_R-CNN_ICCV_2017_paper.pdf" rel="nofollow noreferrer">Mask-RCNN</a> is a notable approach that builds on the <a href="https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf" rel="nofollow noreferrer">Faster R-CNN</a> object detection model, adding a segmentation branch in the per-region network that predicts the mask of the instance. Turns out that this method can be extended to pose estimation. Instance segmentation cares about detecting <em>things</em> objects, like cars, and people.</li>
  </ul>
  <h3>Panoptic Segmentation</h3>
  <p>Panoptic Segmentation unifies stuff and things categories with instances as well. The categories are divided into two classes: stuff (uncountable and/or amorphous objects, like the sky, walls, and road), and things (countable objects, like cars and people). Moreover, ambiguous or out-of-class pixels are assigned a special <em>void label</em>, also each pixel predicted as thing is assigned an instance ID. The <a href="http://openaccess.thecvf.com/content_CVPR_2019/papers/Kirillov_Panoptic_Feature_Pyramid_Networks_CVPR_2019_paper.pdf" rel="nofollow noreferrer">Panoptic FPN</a> builds on Mask R-CNN: it uses two FCNs to predict stuff and things pixels, while a mask FCN predicts the segmentation mask as well. It builds on the idea of <em>Feature Pyramid Network</em> (FPN) - introduced in <a href="https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf" rel="nofollow noreferrer">Faster R-CNN</a> - that rescales and merges extracted features at multiple levels (resolutions), to provide enough information for the semantic segmentation predictions.</p>
  <p>Panoptic segmentation predictions are evaluated using a single unified metric the <em>Panoptic Quality</em> (PQ), that involves two steps:
  <a href="https://i.sstatic.net/JGfNc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JGfNc.png" alt="enter image description here" /></a></p>
  <ol>
  <li><strong>Segment matching:</strong> finds the predicted segmentations that match with the ground-truth (GT). A match occurs only if the IoU between the prediction and the GT is above 0.5, having each GT at most one associated segmentation.</li>
  <li><strong>PQ computation:</strong> each match is split into true positives (TP), false positives (FP), and false negatives (FN) used in the following formula that is not evaluated for void pixels
  <span class="math-container">$$PQ = \frac{\sum_{(p,g)\in TP}IoU(p,g)}{|TP|+\frac12|FP|+\frac12|FN|}$$</span></li>
  </ol>
  <p>To train a panoptic segmentation model is possibile to reuse the dataset that provide <em>both</em> semantic and instance segmentation annotations, like: Cityscapes, ADE20k, and Mapillary Vistas.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-25T22:41:09.773Z
Answers.Text: Are you sure that this Panoptic FPN introduces the idea of "Feature Pyramid Network"? Wasn't this already present in some previous model? It's been many months since I've done anything related to computer vision but I remember these FPNs and they were not introduced, as far as I remember, with this Panoptic FPN (because I don't remember this Panoptic FPN).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-26T06:51:07.910Z
Answers.Text: @nbro You're totally right, thanks for pointing that out! FPN were first introduced in Faster R-CNN. The Panoptic FPN "reuses" that concept. I'll edit that and provide the reference to faster r-cnn, thanks again.
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T18:21:21.320Z
Answers.LastActivityDate: 2023-05-26T06:53:39.853Z
Answers.LastEditDate: 2023-05-26T06:53:39.853Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>In computer vision what is Panoptic Segmentation about? How it relates to Semantic Segmentation? and how it compares to Instance Segmentation?</p>
  
Tags: <deep-learning><computer-vision><semantic-segmentation><instance-segmentation>
Title: What is Panoptic Segmentation?
UserDisplayName: Luca Anzalone
ThreadText: # What is Panoptic Segmentation?
  
  (Asked by: username_0 on 2023-05-04)
  
  In computer vision what is Panoptic Segmentation about? How it relates to Semantic Segmentation? and how it compares to Instance Segmentation?
  
  **Tags:** deep-learning, computer-vision, semantic-segmentation, instance-segmentation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-05-04. Score: 0
  
  [Panoptic Segmentation](https://openaccess.thecvf.com/content_CVPR_2019/papers/Kirillov_Panoptic_Segmentation_CVPR_2019_paper.pdf) (PS) is a computer vision (CV) task that aims at *unifying* both semantic and instance segmentation.
  
  [![enter image description here](https://i.sstatic.net/yHead.jpg)](https://i.sstatic.net/yHead.jpg)
  
  * **Semantic segmentation:** each pixel is labeled with a category ($C$ in total), obtaining a *dense prediction* over an image. The output image has the same resolution of the original one, but only contains the predicted categories of each pixel. If we consider a specific category $c$ in the output image, we obtain the *segmentation mask* that ideally corresponds to the area that all the objects belonging to category $c$ occupy. Usually, the categories $C$ depict *stuff* objects: amorphous regions of same texture and/or material, like the roads and walls. Popular dataset are the PASCAL-VOC2, COCO, ADE20k, and CityScapes. Segmentation models are evaluated with a *generalized IoU* measure over segmentation masks. Differently from object detection, since there is no overlap among pixels there is no need to apply non-maximum suppression to reject multiple predictions. Common neural network architectures are the [FCNs](https://openaccess.thecvf.com/content_cvpr_2015/papers/Long_Fully_Convolutional_Networks_2015_CVPR_paper.pdf), [U-Net](https://arxiv.org/pdf/1505.04597.pdf), and [DeepLab](https://arxiv.org/pdf/1606.00915) based ones.
  * **Instance segmentation:** is about separating different categories at the pixel level into different instances of the same class. Compared to object detection, instance segmentation provides an accurate segmentation of the object (instead of a rectangular region) that localizes it in the space. Each detected instance of the same category is assigned an ID. [Mask-RCNN](http://openaccess.thecvf.com/content_ICCV_2017/papers/He_Mask_R-CNN_ICCV_2017_paper.pdf) is a notable approach that builds on the [Faster R-CNN](https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf) object detection model, adding a segmentation branch in the per-region network that predicts the mask of the instance. Turns out that this method can be extended to pose estimation. Instance segmentation cares about detecting *things* objects, like cars, and people.
  
  ### Panoptic Segmentation
  
  Panoptic Segmentation unifies stuff and things categories with instances as well. The categories are divided into two classes: stuff (uncountable and/or amorphous objects, like the sky, walls, and road), and things (countable objects, like cars and people). Moreover, ambiguous or out-of-class pixels are assigned a special *void label*, also each pixel predicted as thing is assigned an instance ID. The [Panoptic FPN](http://openaccess.thecvf.com/content_CVPR_2019/papers/Kirillov_Panoptic_Feature_Pyramid_Networks_CVPR_2019_paper.pdf) builds on Mask R-CNN: it uses two FCNs to predict stuff and things pixels, while a mask FCN predicts the segmentation mask as well. It builds on the idea of *Feature Pyramid Network* (FPN) - introduced in [Faster R-CNN](https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf) - that rescales and merges extracted features at multiple levels (resolutions), to provide enough information for the semantic segmentation predictions.
  
  Panoptic segmentation predictions are evaluated using a single unified metric the *Panoptic Quality* (PQ), that involves two steps:
  [![enter image description here](https://i.sstatic.net/JGfNc.png)](https://i.sstatic.net/JGfNc.png)
  
  1. **Segment matching:** finds the predicted segmentations that match with the ground-truth (GT). A match occurs only if the IoU between the prediction and the GT is above 0.5, having each GT at most one associated segmentation.
  2. **PQ computation:** each match is split into true positives (TP), false positives (FP), and false negatives (FN) used in the following formula that is not evaluated for void pixels
     $$PQ = \frac{\sum\_{(p,g)\in TP}IoU(p,g)}{|TP|+\frac12|FP|+\frac12|FN|}$$
  
  To train a panoptic segmentation model is possibile to reuse the dataset that provide *both* semantic and instance segmentation annotations, like: Cityscapes, ADE20k, and Mapillary Vistas.
  
  > **Comments:**
  > **username_1**: Are you sure that this Panoptic FPN introduces the idea of "Feature Pyramid Network"? Wasn't this already present in some previous model? It's been many months since I've done anything related to computer vision but I remember these FPNs and they were not introduced, as far as I remember, with this Panoptic FPN (because I don't remember this Panoptic FPN).  
  > **username_0**: @username_1 You're totally right, thanks for pointing that out! FPN were first introduced in Faster R-CNN. The Panoptic FPN "reuses" that concept. I'll edit that and provide the reference to faster r-cnn, thanks again.  
  
  ---
  
  