Consistency regularization describes a class of approaches that have yielded ground breaking results in semi-supervised classification problems. Prior work has established the cluster assumption -under which the data distribution consists of uniform class clusters of samples separated by low density regions -as key to its success. We analyze the problem of semantic segmentation and find that the data distribution does not exhibit low density regions separating classes and offer this as an explanation for why semi-supervised segmentation is a challenging problem. We then identify the conditions that allow consistency regularization to work even without such low-density regions. This allows us to generalize the recently proposed CutMix augmentation technique to a powerful masked variant, CowMix, leading to a successful application of consistency regularization in the semi-supervised semantic segmentation setting and reaching state-of-the-art results in several standard datasets.
# Summary
This paper proposes a method for semi-supervised semantic segmentation. 
The authors tackle this problem through consistency regularization, a successful technique in image classification, that encourages the network to give consistent predictions on unlabeled samples which are perturbed in multiple ways. The authors argue that the cluster assumption (to which effectiveness of consistency regularization has been partially attributed) does not hold in semantic segmentation. Thus, in order to enable class boundaries to become low-density regions and then better guide contrastive regularization, the authors argue that a stronger perturbation must be inserted. To this effect they first propose looking at CutOut and CutMix types of methods. They improved upon them by putting forward a variant of CutMix, coined CowMix, with more degrees of freedom and using flexible masks instead of rectangular ones. CowMix is evaluated on the Cityscapes and PascalVOC 2012 datasets in the semi-supervised regime and showing encouraging results.
# Rating
I find the paper and the advanced ideas of interest for the community and I consider they are novel. I'm currently on the fence between Weak Accept and Weak Reject, mostly due to incomplete evaluations and support for claims made in the introduction regarding the infeasibility of contrastive regularization methods for semantic segmentation. I would be happy to upgrade my rating if authors addressed these concerns.
# Strong points
- The paper is well written and mostly clear with a good coverage and positioning w.r.t. related work. The authors illustrate well the reasoning and the choices they have made. The author provide plenty of ablation studies (e.g., per class statistics) and implementation details, improving significantly the reproducibility of the contribution.
- The flexible masking technique that is advanced here is novel and experimentally seems effective.
- This work is among the few that address semi-supervised semantic segmentation in a non-adversarial manner, so I would give it some novelty credit.
- I appreciate the evaluation protocol of averaging across multiple runs.
# Weak points
## Unclear aspects
- The authors argue that consistency regularization has had little success so far in semantic segmentation problems since low density regions in input data do not align well with class boundaries. It would be useful to provide a reference to this claim or at least validate it experimentally on a large dataset.
- In Figure 1, it is not clear on which features where the distances between patches computed? Is it on raw pixels or intermediate feature maps from a CNN? 
If the distances are made over raw pixels, I find it difficult to make the connection between distances in the pixel space and distances in the class space.
Are the neighbor patches overlapping with the central/query patch?
## Experiments
- The authors compare against other methods on the CamVid dataset. CamVid is a small and relatively limited dataset (~367 images for training from the streets of Cambridge). I'm worried that this dataset might not be enough to conclude and emphasize the benefits of this method over other semi-supervised techniques. For instance CowOut does not seem do be above CutOut, while CutMix has convergence problems and low scores. 
The other experiments on Cityscapes and Pascal VOC are certainly interesting, but the method is compared only against Hung et al. which a different family of methods and the subset baseline (which is useful but not enough). I think this work would benefit from an additional baseline in the style of contrastive regularization methods, e.g. ICT, and eventually CutOut, to support the initial arguments regarding the limitations of these methods in semantic segmentation and respectively the effectiveness of the flexible masks over the rectangular ones in this setup.
# Suggestions for improving the paper:
1) It would be useful to include other semi-supervised baselines, e.g. ICT, and the baseline perturbation CutMix on larger experiments, in order to better emphasize the contributions of this work.
2) Did the authors try the flexible masking on image classification? How is it expected to perform over ICT, MixUp or MixMatch?