This work addresses class-specific object boundary extraction, i.e., retrieving boundary pixels that belong to a class of objects in the given image. Although recent ConvNet-based approaches demonstrate impressive results, we notice that they produce several false-alarms and misdetections when used in real-world applications. We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations, other pixels pose a higher level of difficulties, for instance, region pixels with an appearance similar to the boundaries; or boundary pixels with insignificant edge strengths. Therefore, the training process needs to account for different levels of learning complexity in different regions to overcome false alarms. In this work, we devise a curriculumlearning-based training process for object boundary detection. This multi-stage training process first trains the network at simpler pixels (with sufficient edge strengths) and then at harder pixels in the later stages of the curriculum. We also propose a novel system for object boundary detection that relies on a fully convolutional neural network (FCN) and wavelet decomposition of image frequencies. This system uses high-frequency bands from the wavelet pyramid and augments them to conv features from different layers of FCN. Our ablation studies with contourMNIST dataset, a simulated digit contours from MNIST, demonstrate that this explicit high-frequency augmentation helps the model to converge faster. Our model trained by the proposed curriculum scheme outperforms a state-of-the-art object boundary detection method by a significant margin on a challenging aerial image dataset.
two-stage training scheme for class-specific object boundary estimation. This is a sample result from the model when applied for the problem of extracting road contours from aerial image tiles.
Class-specific object boundary extraction from images is a fundamental problem in Computer Vision (CV). It has been used as a basic module for several applications including object localization [Yu et al. (2018a); Wang et al. (2015)], 3D reconstruction [Lee et al. (2009); Malik & Maydan (1989); ], image generation [Isola et al. (2017); ], multi-modal image alignment Kuse & Shen (2016), and organ feature extraction from medical images [Maninis et al. (2016)]. Inspired from the sweeping success of deep neural networks in several CV fields, recent works [Yu et al. (2017); Acuna et al. (2019); Yu et al. (2018b)] designed ConvNet-based architectures
for object boundary detection and demonstrated impressive results. However, we notice that the results from these methods, as shown in Figure 1b, still suffer from significant false-alarms and misdetections even in regions without any clutter. We hypothesize that although boundary detection is simple at some pixels that are rooted in identifiable high-frequency locations, other pixels pose a higher level of difficulties, for instance, region pixels with an appearance similar to boundarypixels; or boundary pixels with insignificant edge strengths (ex: camouflaged regions). Therefore, the training process needs to account for different levels of learning complexity around different pixels to achieve better performance levels.
In classical CV literature, the different levels of pixel complexities are naturally addressed by decomposing the task into a set of sequential sub-tasks of increasing complexities [Leu & Chen (1988); Lamdan & Wolfson (1988); Kriegman & Ponce (1990); Ramesh (1995)]. Most often, boundary detection problem has been decomposed into three sub-tasks: (a) low-level edge detection such as Canny [Canny (1986)]; (b) semantic tagging/labeling of edge pixels [Prasad et al. (2006)] and (c) edge linking/refining [Jevtić et al. (2009)] in ambiguous regions. These approaches first solve the problem for simpler pixels (with sufficient edge strength) and then reason about harder pixels in the regions with ambiguous or missing evidence. However, with the advent of ConvNets, this classical perspective towards boundary extraction problem has been overlooked. New end-to-end trainable ConvNets have pushed the boundaries of state-of-the-art significantly compared to classical methods. However, we believe that classical multi-stage problem-solving schemes can help to improve the performance of ConvNet models. A parallel machine learning field, Curriculum Learning [Bengio et al. (2009)] also advocates this kind of multi-stage training schemes which train the network with a smoother objective first and later with the target task objective. These schemes are proven to improve the generalization of the models and convergences of training processes in several applications. Motivated by these factors, this work devises a curriculum-learning inspired two-stage training scheme for object boundary extraction that trains the networks for simpler tasks first (subtasks a and b) and then, in the second stage, trains to solve the more complex sub-task (c). Our experimental results on a simulated dataset and a real-world aerial image dataset demonstrate that this systematic training indeed results in better performances.
As mentioned already, the task of predicting object boundaries is mostly rooted in identifiable higher-frequency image locations. We believe that explicit augmentation of high-frequency contents to ConvNet will improve the convergence of training processes. Hence, this work designs a simple fully convolutional network (FCN) that takes in also high-frequency bands of the image along with the RGB input. Here in this work, we choose to use high-frequency coefficients from wavelet decomposition [Stephane (1999)] of the input image and augment them to conv features at different levels. These coefficients encode local features which are vital in representing sharp boundaries. Our empirical results convey that this explicit high-frequency augmentation helps the model to converge faster, especially in the first stage of curriculum learning.
In summary, our contributions in this work are the following:
• A novel two-stage training scheme (inspired from curriculum-learning) to learn classspecific object boundaries.
• A novel ConvNet augmented by high-frequency wavelets.
• A thorough ablation study on a simulated MNIST digit-contour dataset.
• Experiments with a challenging aerial image dataset for road contour extraction • A real-world application of road contour extraction for aligning geo-parcels to aerial imagery.
Related Work: The problem of extracting object boundaries has been extensively studied in both classical and modern literature of CV. Most of the classical methods start with low-level edge detectors and use local/global features to attach semantics to the detected pixels. They, later, use object-level understanding or mid-level Gestalt cues to reason about missing edge links and occluded boundaries. The work by Prasad et al. (2006) used local texture patterns and a linear SVM classifier to classify edge pixels given by Canny edge detector. The work by Mairal et al. (2008) reasoned on low-level edges, but learned dictionaries on multiscale RGB patches with sparse coding and used the reconstruction error curves as features for a linear logistic classifier. The work of Hariharan et al. (2011) proposed a detector that combines low-level edge detections and semantic outputs from pre-trained object detectors to localize class-specific contours.
Several recent works Yang et al. (2016); Yu et al. (2017) adopted fully convolutional networks (FCN) for the task of semantic boundary extraction. The work by Yang et al. (2016) proposed a FCN-based encoder-decoder architecture for object contour detection. The work of Bertasius et al. (2015) first used VGG-based network to locate binary semantic edges and then used deep semantic segmentation networks to obtain category labels. More recently, the work of Yu et al. (2017) proposed an FCN architecture, CASENet, with a novel shared concatenation scheme that fuses low-level features with higher conv layer features. The shared concatenation replicates lower layer features to separately concatenate each channel of the class activation map in the final layer. Then, a K-grouped 1 × 1 conv is performed on fused features to generate a semantic boundary map with K channels, in which the k-th channel represents the edge map for the k-th category. A few recent works Acuna et al. (2019); Yu et al. (2018b) integrated an alignment module into the network to account for noise in the contour labels. Most of these networks are trained in an end-to-end manner, using crossentropy-based objective functions. These objectives treat all boundary pixels equally irrespective of the complexity around them. Unlike existing methods, we use explicit high-frequency augmentation to ConvNet and train it in a curriculum learning scheme that accounts for different levels of pixel complexities with two stages.
2 THE PROPOSED CURRICULUM LEARNING SCHEME  (2012)] start with a smoothed objective function and gradually move to less smoothed functions. In other terms, these methods consider a class of objective functions that can be expressed as,
where C o (θ) is smoother or simple objective and C t (θ) is the target objective we wish to optimize. There are several ways to choose C o ; it can either be the same loss function as C t but solving the task on simpler examples, or be a proxy task simpler than the target task. In general, λ is a (binary) variable that takes values zero or one. It is set to zero initially and later increases to one. The epoch where it changes from zero to one is referred to as the switch epoch.
Curriculum Learning in CV: CL-inspired training schemes are recently gaining attention in CV fields. Recently some popular CV architectures leveraged CL based schemes to improve model generalization and training stabilities. In FlowNet 2.0 [Ilg et al. (2017)] for optical flow prediction, simpler training data are fed into the network first and then the more difficult dataset. The object detection framework of  first trains simpler networks (proposal, refiner-nets) and then trains the final output-net in the end. Here we propose a two-stage CL scheme for object boundary detection methods.
The proposed CL based training scheme for learning Object Boundaries: ConvNets for classspecific object boundary detection are in general trained with multi-label cross entropy-based objectives [Yu et al. (2017)].
where θ denotes the weights of the network; and p and k represent indices of pixel and class labels respectively.Ŷ and Y represent prediction and groundtruth label maps. β is the percentage of non-edge pixels in the image to account for skewness of sample numbers [Yu et al. (2017)]. This objective treats all the contour pixels equally and does not account for the complexity of the task around them. Here, we consider this as the target objective function, C t , that we wish to optimize.
We start the training, however, with a simpler task C o . We believe the pixels with strong edge strength are easy to localize and semantically identify. Hence, we propose to solve the task around those pixels in the first stage. We take element-wise multiplication between canny edge map of the input and dilated groundtruth label to prepare supervisory signal Z for this stage.
where E I is the Canny edge map of image I and Y D the dilated groundtruth map. This is as shown in Figure 2e. The objective function for this stage becomes;
Since we use a dilated version of GT in preparing Z, it also contains some non-object contour pixels. However, these might be refined in the second stage of CL, when trained with Y (Eq 2). Hence, the CL objective function in Eq 1 uses Eq 4 and Eq 2 as initial and target objective functions respectively. In the CL training scheme, we set the switch epoch as T /2, where T is the total number of training epochs.
Summary: The suggest two improvements to boundary detection models: (1) a curriculum learning approach, and (2) augmenting CNNs with features derived from a wavelet transform. For (1), they train half of the epochs with a target boundary that is the intersection between a Canny edge filter and the dilated groundtruth. The second half of epochs is with the normal groundtruth. For (2), they compute multiscale wavelet transforms, and combine it with each scale of CNN features. They find on a toy MNIST example that the wavelet transform doesn’t impact results very much and curriculum learning seems to provide some gains. On the Aerial Road Contours dataset, they find an improvement of ~15% mAP over the prior baseline (CASENet).
I have several concerns with this work:
* The idea of using wavelet transforms to augment CNNs has been more thoroughly explored in prior work (e.g., see [1]).
* No comparison to existing SOTA segmentation models (e.g., [2]). These semantic / instance segmentation models can easily be adapted to the task of boundary detection. I suspect the baseline here is weak.
* Section 6 is severely unfinished. The explanation is sparse and there are no quantitative results -- just the output of the model overlaid on one example.
* The choice of curriculum learning task is arbitrary, and there are no ablations explaining why this is a reasonable task. For example, what about random subsets of pixels? At the moment, it offers no insight for practitioners.
* There are no ablations for the Aerial Road Contours experiments. This seems necessary because it is the only realistic dataset evaluated in this work. The MNIST experimental results appear qualitatively different from the Contours experiment. For example, they show that wavelet features do not make much of a difference, but does it make a difference for Contours?
Altogether, this work unfortunately offers few insights to vision practitioners, let alone general practitioners. Substantial work needs to be devoted to expanding experimental coverage. 
[1] Wavelet Convolutional Neural Networks. Shin Fujieda, Kohei Takayama, Toshiya Hachisuka
[2] TensorMask: A Foundation for Dense Object Segmentation. Xinlei Chen, Ross Girshick, Kaiming He, Piotr Dollár