Learning-based approaches for semantic segmentation have two inherent challenges. First, acquiring pixel-wise labels is expensive and time-consuming. Second, realistic segmentation datasets are highly unbalanced: some categories are much more abundant than others, biasing the performance to the most represented ones. In this paper, we are interested in focusing human labelling effort on a small subset of a larger pool of data, minimizing this effort while maximizing performance of a segmentation model on a hold-out set. We present a new active learning strategy for semantic segmentation based on deep reinforcement learning (RL). An agent learns a policy to select a subset of small informative image regions -opposed to entire images -to be labeled, from a pool of unlabeled data. The region selection decision is made based on predictions and uncertainties of the segmentation model being trained. Our method proposes a new modification of the deep Q-network (DQN) formulation for active learning, adapting it to the large-scale nature of semantic segmentation problems. We test the proof of concept in CamVid and provide results in the large-scale dataset Cityscapes. On Cityscapes, our deep RL region-based DQN approach requires roughly 30% less additional labeled data than our most competitive baseline to reach the same performance. Moreover, we find that our method asks for more labels of under-represented categories compared to the baselines, improving their performance and helping to mitigate class imbalance. 1 1 INTRODUCTION Semantic segmentation, the task of labelling an image pixel-by-pixel with the category it belongs to, is critical for a variety of applications such as autonomous driving (Müller et al., 2018;Wang & Pan, 2018), robot manipulation (Schwarz et al., 2018), embodied question answering (Yu et al., 2019) and biomedical image analysis (Ronneberger et al., 2015). Convolutional neural networks (Lecun et al.,  1998)-based methods have achieved excellent results on large-scale supervised semantic segmentation, in which we assume pixel-level annotations are available (Farabet et al., 2013; Pinheiro & Collobert,  2014; Long et al., 2015). For such models to work, however, they need a large amount of pixel-level annotations that may require costly human labor (Cordts et al., 2016;Bearman et al., 2016).
Current semantic segmentation datasets have pixel-wise annotations for each image. This standard approach has two important issues: (i) pixel-level labelling is extremely time consuming. For example, annotation and quality control required more than 1.5h per image (on average) on Cityscapes (Cordts et al., 2016), a popular dataset used for benchmarking semantic segmentation methods. (ii) Class imbalance in the data is typically extreme. Certain categories (such as 'building' or 'sky') can appear with two orders of magnitude more frequently than others (e.g. 'pedestrian' or 'bicycle'). This can lead to undesired biases and performance properties for learned models. This is specially relevant when we want to collect annotated data with a human in the loop to create a new dataset or to add more labeled data to an existing one. We can tackle the aforementioned problems by selecting, in an efficient and effective way, which regions of the images should be labeled next. Active learning (AL) is a well-established field that studies precisely this: selecting the most informative samples to label so that a learning algorithm will perform better with less data than a non-selective approach, such as labelling the entire collection of data. Active learning 1 Code will be available upon acceptance.
# Summary #
The paper works on active learning for semantic segmentation, aiming to annotate as few "blocks/patches" as possible while training a strong model. The authors proposed to learn a query policy via Q learning, and design states and actions specifically for segmentation. The experimental results show that the learned policy can attend to informative patches and rare classes to learn a model faster and efficiently.
# Strength #
S1. The paper is well-motivated; the references are quite sufficient.
S2. The paper clearly states the challenges when applying RL algorithms like Q-learning to image segmentation, which should serve as good guidance for other future work.
#Weakness/comments#
W1. The writing of the technical part can be strengthened.  The authors deferred the state and action design entirely to the supplementary, while they are the main contributions to the paper.
W2. The proposed algorithms seem to be highly time-consuming. The actions require pairwise comparison, and at every step, the models need to evaluate all the validation images to get the reward.
W3. If I understand correctly, the authors use part of the training data D_T, D_S (of an existing dataset) together with the validation data D_R to learn the policy, and then use the learned policy to select patches from the remaining training data D_V to train the segmentation model. I have two questions.
1) The labeled data involved in policy training is indeed quite large (validation plus part of the training, D_S + D_T + D_R). Does it mean that to learn a good active learning policy we indeed need a large number of labeled data?
2) Since (D_S, D_T, and D_R) are used to learn the policy, they should be treated as available training data for segmentation that all the compared algorithms can use without spending the budget. In other words, all the compared algorithms (U, H, B) should use those data to fine-tune a pre-train segmentation network before they start to acquire data from D_V. It would be great if the authors can clarify this.
W4. In applying the policy for selecting patches from D_V, do the authors update the model once on the selected patches, or do the authors train with them for multiple iterations together with other previous selected patches? Since deep neural nets are known to forget what has been learned (i.e., catastrophic forgetting), it's better if the author could clarify this.
W5. The authors include an upper bound in Fig. 4; however, I didn't find the explanation. Why the proposed methods can outperform the upper bound with all the training data, even with only 24% of data?
#Rebuttal#
Please discuss W1-W5.
- Annotating an entire image is definitely easier to annotators than annotating patches. Could the authors discuss how to design an active learning algorithm by selecting informative images to annotate, and maybe compare to such a method?
- Can the authors discuss Figure 3 more? As H is based on maximum entropy, why is it outperformed by the proposed method? 
- There is no explicit mechanism to prevent that the k actions select similar patches. Can the authors provide more discussion?
# Post rebuttal
The authors responded to most of my concerns. I'd like the authors to incorporate all their responses into the manuscript or the appendix so that future readers can better understand the concepts and details.  I would like to raise the score to borderline (4 or 5). I modified my scores to weak accept (6) since there is no option in between.
One concern I still have is W3. 2). Given only 360 images are available, it might be inappropriate to use D_T + D_S (roughly 160 images?) to pre-train baseline models and then use D_R with "200" images for validation (early stopping). It will be more appropriate to use, for example, 70% of 360 images for training. This is supported by that many modern datasets use a much larger training set than the validation set. Therefore, I would highly suggest the authors redoing the baseline methods; otherwise, future work that re-splits the data (this is totally valid!) from the 360 images might easily achieve higher accuracy.
The authors must also reorganize the paper, taking W1 into account.