The need for labour intensive pixel-wise annotation is a major limitation of many fully supervised learning methods for image segmentation. In this paper, we propose a deep convolutional neural network for multi-class segmentation that circumvents this problem by being trainable on coarse data labels combined with only a very small number of images with pixel-wise annotations. We call this new labelling strategy 'lazy' labels. Image segmentation is then stratified into three connected tasks: rough detection of class instances, separation of wrongly connected objects without a clear boundary, and pixel-wise segmentation to find the accurate boundaries of each object. These problems are integrated into a multitask learning framework and the model is trained end-to-end in a semi-supervised fashion. The method is demonstrated on two segmentation datasets, including food microscopy images and histology images of tissues respectively. We show that the model gives accurate segmentation results even if exact boundary labels are missing for a majority of the annotated data. This allows more flexibility and efficiency for training deep neural networks that are data hungry in a practical setting where manual annotation is expensive, by collecting more lazy (rough) annotations than precisely segmented images.
The submission presents a neural network for multi-task learning on sets of labeled data that are largely weakly supervised (in this case, partially segmented instances), augmented by comparatively fewer fully supervised annotations.
The multiple tasks are designed to make use of both the weak as well as as full (‘strong’) labels, such that performance on fully annotated machine-generated output is improved.
As noted in the related work section (Section 2), multi-task methods aim to use benefits from underlying common information that may be ignored in a single-task setting. The network presented here is quite similar to most of these multi-task approaches: a common feature encoder, and partially distinct feature decoding and classification parts.
The (minor) novelty mainly comes from the distinct types of weak/strong annotation data fed here: instance scribbles, boundary scribbles, and (some or few) full segmentations. 
The submission is overall well written and provides sufficient clarity and a good overview of the approach.
Section 3 presents a probabilistic decomposition of the proposed architecture. With some fairly standard assumptions and simplifications, the loss in Eq. 3 becomes rather straightforward (weighted cross entropy)
The actual network architecture described in Section 3.2 takes a standard U-Net as a starting point and modifies it in a fairly targeted way for the different expected types of annotations. These annotations (Section 3.3) are cheaper than full labels on a same-size dataset; it is not completely clear, however, if the mentioned scribbles need to capture each instance in the training set, or if some can also be left out. Without this being explicitly mentioned, I will assume the former.
The experimental evaluation is done reasonably well, although I am not familiar with any of the presented data sets. The SES set seems to be specific to the submission, while the H&E data set has been used at least one other relevant publication (Zhang et al.). My main issue here is that at least on the SES set, which does not seem to be that large, the score difference is not that big, so dataset bias could play some part (which is unproven, but so is the opposite).
Experimental evaluation does not leave the low-number-of-classes regime, and I’m left wondering how the method might compare on a semantically much richer data set, e.g. Cityscapes. Finally, unmodified U-Net is by now a rather venerable baseline, so I’m also wondering how the proposed multi-task learning could be used in other (more recent) architectures, i.e. whether the idea can be generalized sufficiently.
While I think the ideas per se have relatively minor novelty, the combination seems novel to me, and that might warrant publication.