Large-scale ground truth data sets are of crucial importance for deep learning based segmentation models, but annotating per-pixel masks is prohibitively time consuming. In this paper, we investigate interactive graph-based segmentation algorithms that enforce connectivity. To be more precise, we introduce an instanceaware heuristic of a discrete Potts model, and a class-aware Integer Linear Programming (ILP) formulation that ensures global optimum. Both algorithms can take RGB, or utilize the feature maps from any DCNN, whether trained on the target dataset or not, as input. We present competitive semantic (and panoptic) segmentation results on the PASCAL VOC 2012 and Cityscapes dataset given initial scribbles. We also demonstrate that our interactive approach can reach 90.6% mIoU on VOC validation set with an overhead of just 3 correction scribbles. They are thus suitable for interactive annotation on new or existing datasets, or can be used inside any weakly supervised learning framework on new datasets.
Summary:
- key problem: efficiently leveraging scribbles as interactive supervision (at test time) for panoptic segmentation;
- contributions: 1) two algorithms leveraging scribbles via a superpixel connectivity constraint (one class-agnostic local diffusion heuristic, one class-aware with a MRF formulation), 2) experiments on PASCAL VOC 2012 and Cityscapes showing that both methods i) can achieve good performance without training data (using RGB values or pretrained features), ii) can improve the performance of a fully supervised baseline when using its probability maps as representation, and iii) can significantly improve performance beyond the state of the art on PASCAL when used interactively (90% mIoU with 3 rounds of corrective scribbles).
Recommendation: weak reject
Key reason 1: unclear novelty and relevance to ICLR.
- The paper proposes to apply two existing algorithms (Nguyen & Brown 2015, Rempfler et 2016) to a new task (interactive panoptic segmentation): what is the claimed novelty? What is specific to panoptic segmentation vs semantic or instance segmentation? Could the difference with related work in Section 2 be discussed more precisely?
- Furthermore, there seems to be no learning (representation or otherwise) involved in this submission. The paper mentions potential applications to weakly-supervised learning in Section 5, but it does not provide clear insights into what would be the benefits in terms of representation learning (vs. RGB, pre-trained features, or probability maps).
- Overall, this paper might be more tailored for a Computer Vision venue like CVPR.
Key reason 2: lack of sensitivity / robustness analysis.
- The scribbles are "simulated" using morphological operations on the ground truth (A.2, A.3): does this lead to realistic scribbles? Figure 3 (which is unclear) shows that the "scribbles" might be precise outlines or contours, which are very different than the expected scribbles illustrated in Figure 2. Contours provide much stronger information for segmentation, and are much more likely to effectively leverage the connectivity prior (esp. with the diffusion heuristic), but are they really scribbles / cheap supervision?
- What is the importance of the superpixel coverage by scribbles or missing scribbles or the location of scribbles relative to segment boundaries? What are the impact of realistic deviations from the expected scribble policy that are likely to happen in practice? Measuring sensitivity to different types of noise (by perturbing / dropping out scribbles) seems important to assess the practical usefulness and robustness of the method.
- PASCAL VOC and Cityscapes are small datasets. Experiments on bigger more recent ones like Mapillary Vistas and COCO are becoming the standard protocol in the instance/semantic/panoptic segmentation community. How would this method fare on those much more challenging datasets? What are the benefits of the proposed interactive methods in terms of scalability?
Additional Feedback:
- Fig. 4 is too low resolution / blurry;
- typos: "tarining set", "weekly supervised".
## Update following the rebuttal
Thanks to the authors for their replies. Sadly, my concerns are only answered at a high-level, and the consensus among reviewers is clear. Hence I confirm my rating to reject. I hope the feedback provided above will assist the authors in improving the work or finding a more suitable venue.