summaries: This papers proposed a solution to the problem of disease density estimation using satellite scene images. One common challenge in this type of applications is having a high intra-class diversity and a high inter-class similarity. The solution proposed by the authors is based on the use of siamese networks to extract features from pairs of neighbouring images, and merge the features only if they are similar. The authors claim that this approach alleviate the need of a post-classification smoothing.

summaries: The paper proposes to do a coupled inference over pairs of geographically close images instead of a single image for satellite imagery. The coupling is done with an average pooling of the feature vectors when the neighbouring patches are detected to be similar enough based on a threshold on the L2 distance of these features. The method is applied to tasks of estimating crowding population, and diseases density, from satellite images.

abstractText: High intra-class diversity and inter-class similarity is a characteristic of remote sensing scene image data sets currently posing significant difficulty for deep learning algorithms on classification tasks. To improve accuracy, post-classification methods have been proposed for smoothing results of model predictions. However, those approaches require an additional neural network to perform the smoothing operation, which adds overhead to the task. We propose an approach that involves learning deep features directly over neighboring scene images without requiring use of a cleanup model. Our approach utilizes a siamese network to improve the discriminative power of convolutional neural networks on a pair of neighboring scene images. It then exploits semantic coherence between this pair to enrich the feature vector of the image for which we want to predict a label. Empirical results show that this approach provides a viable alternative to existing methods. For example, our model improved prediction accuracy by 1 percentage point and dropped the mean squared error value by 0.02 over the baseline, on a disease density estimation task. These performance gains are comparable with results from existing post-classification methods, moreover without implementation overheads.

authors:
Adrian Albert
Jasleen Kaur
Marta C. Gonzalez

title: Using convolutional networks and satellite imagery to identify patterns in urban environments at a large scale

venue: SigKDD

authors:
Pierre Baldi
Yves Chauvin

title: Neural networks for fingerprint recognition

venue: Neural Comput.,

authors:
Horst Bischof
Wolfger Schneider
Axel J. Pinz

title: Multispectral classification of landsat-images using neural networks

venue: IEEE Transactions on Geoscience and Remote Sensing,

authors:
Jane Bromley
Isabelle Guyon
Yann LeCun
Eduard Säckinger
Roopak Shah

title: Signature verification using a ”siamese” time delay neural network

venue: In Proceedings of the 6th International Conference on Neural Information Processing Systems,

authors:
Marco Castelluccio
Giovanni Poggi
Carlo Sansone
Luisa Verdoliva

title: Land use classification in remote sensing images by convolutional neural networks

authors:
Nitech V. Chawla
Kevin W. Bowyer
Lawrence O. Hall
Philip W. Kegelmeyer

title: Smote: Synthetic minority over-sampling technique

venue: Journal of Artificial Intelligence Research,

authors:
Gong Cheng
Junwei Han
Xiaoqiang Lu

title: Remote sensing image scene classification: Benchmark and state-of-the-art

venue: Proceedings of the IEEE,

authors:
Gong Cheng
Ceyuan Yang
Xiwen Yao
Lei Guo
Junwei Han

title: When deep learning meets metric learning: Remote sensing image scene classification via learning discriminative cnns

title: Xception: Deep learning with depthwise separable convolutions

authors:
Kaiming He
Xiangyu Zhang
Shaoqing Ren
Jian Sun

title: Deep residual learning for image recognition. in 2016 ieee conference on computer vision and pattern recognition

authors:
Xuning Liu
Yong Zhou
Jiaqi Zhao
Rui Yao
Bng Liu
Yi Zheng

title: Siamese convolutional neural networks for remote sensing scene classification

venue: IEEE Geoscience and Remote Sensing Letters,

title: Machine learning for aerial image labeling

venue: PhD thesis, University of Toronto.,

authors:
Volodymyr Mnih
Geofrey Hinton

title: Learning to detect roads in high-resolution aerial images

venue: Proceedings of the 11th European Conference on Computer Vision (ECCV),

authors:
Keiller Nogueira
Otávio Augusto Bizetto Penatti
Jefersson Alex dos Santos

title: Towards better exploiting convolutional neural networks for remote sensing scene classification

authors:
Caleb Robinson
Fred Hohman
Bistra Dilkina

title: A deep learning approach for population estimation from satellite

venue: imagery. CoRR,

authors:
Rahman Sanya
Ernest Mwebaze

title: Mapping spatial housing patterns using deep neural networks and remote sensing data

venue: In Neural Information Processing Systems Conference

heading: 1 INTRODUCTION

text:
Remote sensing scene image analysis is emerging as an important area of research for application of deep learning algorithms. Application areas include land-use land-cover analysis, urban planning, and natural disaster detection. A deep learning task for labeling a scene image is typically formulated as conditional probability of the form in Eq. 1 Liu et al. (2019),Albert et al. (2017), Nogueira et al. (2016), Castelluccio et al. (2015), Mnih (2013), Mnih & Hinton (2010),
P (li|si) (1)
where li is label for image patch si. This formulation is sufficient for problems where spatial situatedness of a scene, which embodies knowledge of semantic likeness between neighborhoods in the geophysical world, is not important. However, for problems which require knowledge of neighborhood the formulation in Eq. 1 becomes inadequate. An example of such a problem would be estimating disease density for a small geographical region of interest, in which case the probability of label l is likely to depend on the labels for neighboring regions due to semantic coherence among them.
The problem of how to improve model prediction by leveraging semantic coherence among neighboring scene images has previously been considered in the literature. Previous studies consider the problem as a post-classification task. For example, Bischof et al. (1992) used a second classifier to do pixel smoothing to refine predictions made by another classifier. Based on a 5x5 window, a filter assigns pixels to the majority class if it had been assigned a different class. In Mnih (2013), a post-processing architecture is suggested for incorporating structure into image patch prediction. It involves stacking neural networks (NN) such that the output from a previous one becomes input for the next. Idea is for each network to clean up predictions of previous one in order to progressively
improve overall accuracy. While improved model performance was achieved by these methods, they have overhead of performing same classification task in at least two stages. In other words, you need a minimum of two NN to perform the same classification task.
Unlike post-classification methods, this work considers the problem of improving model accuracy on scene images by exploiting knowledge of neighboring scenes as part of the model training process. We make the assumption that l is conditionally co-dependent on information embedded in scene image i and in other similar, neighboring image j such that the problem is formulated as probability of the form in Eq. 2,
P (li|Si,Sj) = Πmi=1P (li|si, sj) (2)
where sj is image for a neighboring tile that is most similar to index tile i andP (li|Si,Sj) is observed probability distribution.
We used Convolutional Neural Networks (CNN) for modeling the observed probability distribution in Eq. 2. A network architecture is proposed for training our model consisting of four components: a siamese sub-network, a similarity metric learning component, a convolutional network, and a decision layer. The siamese sub-network takes two neighboring scene images as input and extracts features from each. The similarity learning component evaluates how similar the input images are, using the extracted features. If the two input images are found to be similar the convolutional network learns additional features based on the merged feature vector, otherwise those from the index tile are used alone. We implemented the decision layer to perform classification or regression. A baseline model was implemented that takes a single image, the index tile i, as input.
Empirical results show the proposed model consistently outperforms the baseline. In addition to improving predictive performance with a relatively small training set, our model is fast to train since it uses a pre-trained model for the siamese sub-network. Furthermore, it does not require another NN to smooth out its predictions as is the case with post-classification approaches, while achieving comparable performance gain. In summary,our contributions include the following.
1. We propose an approach for training a probabilistic deep learning model to improve prediction accuracy by exploiting semantic coherence between neighboring tiles in aerial scene images. A CNN architecture is suggested for this purpose.
2. We provide empirical evidence that demonstrates the viability of this approach on a disease density estimation task.
3. Lastly, we discovered an important limitation of the synthetic minority over-sampling technique (SMOTE). This method fails when used for oversampling an under-represented class whereby knowledge of spatial proximity between scene image data points must be preserved, an important requirement under the framework of learning deep features over neighboring scene images introduced in this work.

heading: 2 RELATED WORK

text: Remote sensing scene images represent a difficult problem for deep learning algorithms due to high intra-class diversity and inter-class similarity (Cheng et al., 2017). Previous efforts to address this problem is presented first, followed by recent use of siamese networks to improve discriminative ability of CNN.

heading: 2.1 IMPROVING MODEL ACCURACY BY EXPLOITING COHERENCE IN SCENE IMAGES

text:
One of the earliest works aimed at improving model accuracy by exploiting semantic coherence between neighboring images is by Bischof et al. (1992). The problem addressed there is ’salt and pepper’ noise in classified satellite images. The authors employ a multi-layer perceptron (MLP) consisting of five hidden units to classify low resolution (30m) satellite scene images. Their proposed network has four output units corresponding to four land-use classes. The sigmoid activation function was used with both hidden and output units. The network had seven input units, each corresponding to one of seven Landsat TM spectral bands. It generates two separate channels when
classifying an input image: one for class prediction result and the other representing how confident the classification is. A post-classification smoothing operation is then performed using a two-layer NN whose input is the classified image within a 5 x 5 window. For each pixel, the predicted class and confidence information are used as input to decide its new class belonging by applying a majority filter. Gain in classification accuracy was 2.9 percent for the network whose input data was enhanced with texture information (and 4.9 percent for the one without enhancement).
Mnih (2013) introduce indirect dependencies between outputs of a model by using knowledge of structure (e.g. shape) to resolve the issue of disconnected blobs and holes or gaps in predicted building and road network maps. The architecture used consists of stacking NN, each using as input the outputs of a previous network. Precisely, let M̂0 be the map predicted by a model. The ith level cleanup NN (fi) takes a ws x ws patch of ˆMi−1 and outputs a wm x wm patch of M̂i. The model fi is trained by minimizing the negative log likelihood on patches of the observed map M̂ , just like other NN in the architecture. Thus, the ith cleanup NN improves predictions of the previous NN fi−1. The authors extended this idea to Conditional Random Fields (CRF) i.e. pairwise lattice CRF to introduce explicit dependencies between pairs of neighboring pixels. For both NN and CRF, an average gain of 0.0195 was achieved in precision and recall break-even points.
The above works demonstrate that post-classification smoothing operations can improve performance of deep learning models. However, a draw back of those approaches is the necessary requirement to implement additional models to perform smoothing operations. Therefore, approaches that sidestep this requirement would be a welcome alternative.

heading: 2.2 SIAMESE NEURAL NETWORKS

text:
Another innovation that inspired the current work is siamese neural networks (SNN) introduced by Bromley et al. (1993) and Baldi & Chauvin (1993). A SNN architecture consists of two identical (shared weight) sub-networks merged by the same function at their outputs. During training each sub-network extracts features from one of two concurrent inputs, after which the joining function evaluates whether the input pair is similar based on a distance metric, such as cosine of angle between the pair of feature vectors. Output from the SNN is a decision score e.g., 1 for similar image pair, 0 otherwise.
SNN have recently been used for remote sensing scene image understanding. Liu et al. (2019) used a siamese architecture to improve discriminative ability of CNN for satellite scene image classification. The architecture consists of two identical CNN models, 3 additional convolutional layers, and one square non-parametric layer. Each branch of the siamese network is responsible for extracting features from each dual input image, giving feature vectors f1 and f2. Two of the convolutional layers take these feature vectors as input and perform additional learning before a softmax function predicts the label for each input image. On the other hand, the square layer takes f1 and f2 as input and outputs the tensor fs = (f1 − f2)2 which is used to measure similarity between the input pair. The model thus produces three outputs: predicted label for each input image and a score for their similarity.
More precisely, feature discrimination enhancement is achieved in the square layer by imposing a metric learning regularization term that minimizes Euclidean distance between a similar image pair and maximizes that for a dissimilar pair (Eq. 3),
D(xi, xj) = ||f1 − f2||22 (3)
where D(xi, xj) is distance between a pair of input images. A margin τ is set to separate as far apart as possible, the similar image pairs from dissimilar ones in feature space. If (xi, xj) is from the same scene class the distance between them is less than τ , otherwise it is greater than τ . Let label for training pair (xi, xj) be (yi, yj), respectively. The formulation for Euclidean distance between them is given by (Eq. 4),
{ D(xi, xj) < τ, yi = yj D(xi, xj) > τ, yi 6= yj
(4)
This model only minimizes feature distance between similar input pairs through the margin τ to achieve feature discrimination.
Therefore, the model in Liu et al. (2019) optimizes for two objective functions: a distance loss function and a regularization term as shown in Eq. 5,
Dist(f1, f2) = ∑ i,j D(xi, xj), D(xi, xj) < τ (5)
Of the three pre-trained models experimented with (AlexNet, VGG-16, and ResNet-50) highest feature discrimination accuracy gain (1.14 percentage point) was achieved with a Siamese AlexNet model on the challenging NWPU-RESIC45 (Cheng et al., 2017) benchmark data set. Cheng et al. (2018) reported a 1.53 percentage point gain in accuracy on the same data set.

heading: 3 LEARNING DEEP FEATURES OVER NEIGHBORING SCENE IMAGES

heading: 3.1 OVERVIEW

text: An end-to-end pipeline is proposed consisting of four components: a siamese sub-network, similarity metric learning function, convolutional network, and decision layer. The siamese sub-network takes a pair of scene images si (tile we want to predict a label for) and sj (other tile neighboring si) as input and extracts features from each, outputting feature vectors fi and fj , respectively. The similarity function takes as input the two feature vectors and learns a similarity metric between them. If the feature vector pair (fi, fj) is found to be similar they are merged and the resulting feature vector used to train the convolutional network, otherwise training is done using (fi) alone. Output from the convolutional network is passed as input to the decision layer to predict a label (or to regress a numerical value) for image patch si. The proposed and baseline architectures are shown in Figures 1a and 1b, respectively.

heading: 3.2 ARCHITECTURE DESCRIPTION

heading: 3.2.1 SIAMESE SUB-NETWORK

text:
To implement the siamese sub-network we explored two state-of-the-art pre-trained CNN models i.e. Xception net Chollet (2016) and ResNet-50 He et al. (2016) in a transfer learning (fine-tuning) strategy since we had a small training set. The architecture of Xception model is based on separating learning of space-wise and channel-wise feature representations. The result is better performance
over architectures such as Inception V3, ResNet-152, and VGG-16 in terms of computational speed and classification accuracy. On the other hand, ResNet architectures address the problem of vanishing/exploding gradients in deep neural network architectures by inserting shortcut connections between convolutional blocks. The ResNet model has benefit of enabling deeper architectures to be utilized in CNN models since these provide higher accuracy rates than shallow architectures.

heading: 3.2.2 SIMILARITY METRIC LEARNING

text:
The similarity function takes as input feature vector (fi, fj) to evaluate how similar the input image pair is. This unit outputs one of a binary decision score i.e neighboring input pair (si, sj) is similar (1) or dissimilar (0). The optimization function used here works by minimizing feature distance between similar images. Following work in Liu et al. (2019), given a pair of input image si and sj the distance between them is calculated using Eq. 3. A margin τ separates similar image features from dissimilar ones such that D(fi, fj) < τ for a similar pair, D(fi, fj) > τ otherwise. We also optimized for two objective functions as defined in Eq. 5.
An important question to consider is how to identify a neighboring image that is semantically closest in likeness to index tile i, over which to co-learn features. For any image patch in the middle region of a w x h scene image S where w≥3 and h≥3, there will be eight such tiles to choose from, considering four adjacent and diagonal neighbors. For edge tiles there will be five while for corner tiles there will be three neighbors, Figure 2 in Appendix.
We can determine neighbors of the tile at location i by specifying the maximum Harversine distance within which the centroid coordinates of all neighbors will lie. Take as an example tiles of size 250m2 in a 3 x 3 scene image.The centroid of all eight neighbors of the center tile would fall within a maximum distance of 350m from its own centroid. We can evaluate how similar tile at i is to its neighbor at location j using Euclidean distance. If at least one is similar, we select the neighbor with least distance to be most similar. Given that D(si, sj) < τ for images from same scene class, a neighboring image is most similar to si if it has the smallest distance Eq. 6,
ŝi = arg min si (si, sj) (6)
where ŝi is neighboring image most similar to index tile at location i.
In the current analysis however, we consider only the preceding neighbor to tile at location i (image si−1) for similarity analysis. The loss function used is binary cross-entropy, defined in Eq. 7.
L(W, b) = − N∑ i=1 yi × log ŷi + ( 1− yi ) × log ( 1− ŷi ) (7)

heading: 3.2.3 CONVOLUTIONAL NETWORK

text: This network is trained using features resulting from merging feature vector pair (fi, fj) if respective input image pair (si, sj) is found to be similar and it outputs feature vector fm(fi, fj). If the input pair is dissimilar, only the feature vector fi is fed to the convolutional network, which outputs fi(fi). Our implementation used three dense layers in the convolutional network. Merge operation used is feature averaging.

heading: 3.2.4 DECISION LAYER

text:
For a classification task we learn the parameters (weights) of the CNN by minimizing the negative log likelihood of the training data under our model. For the multi-class problem considered in this work the negative log likelihood under the model in Eq. 2 assumes the form of a cross-entropy between the probability distribution for observed labels l and the predicted label probabilities ŷ (Eq. 8),
L(W, b) = − N∑ i=1 M∑ j=1 yi × log ŷi (8)
where M, the number of classes, is 3.
The outer sum of objective function L is over all training samples. Stochastic gradient descent with mini-batches is used for optimizing L. Our implementation for the current problem uses a fully connected network (FCN) with a 3-way softmax classifier.
Taken as a regression task, we optimized for the mean absolute error (MAE) loss and mean squared error (MSE). The form of MAE we used is given by Eq. 9,
L(W, b) = 1
2N N∑ i=1 M∑ j=1 ∣∣∣∣ŷi − yi∣∣∣∣ (9) where M, the number of features to be predicted, is 2. For MSE the loss function is given by Eq. 10,
L(W, b) = 1
N N∑ i=1 1 M M∑ i=1 ( ŷi − yi )2 (10)

heading: 4 EXPERIMENTS

text: We applied our method to the task of estimating disease density from satellite scene imagery. Below we describe data sets and methods used.

heading: 4.1 METHODS AND DATASET

text:
Dataset
A number of data sets were used in our experiments. The epidemic data consists of monthly disease case counts aggregated by sub-county for year 2015. We dis-aggregated the data to 250m2 grids, our geospatial unit of analysis, using population data from Facebook & CIESIN (2016). The latter data gives estimate of people living in a 30m2 grid based on recent census data and high resolution (0.5m2) satellite imagery. The data was up-sampled to 250m2 grid to be consistent with our spatial unit of analysis. The population data was used to create a weighting scheme for dis-aggregating epidemic data. The building (housing) concentration data consists of satellite imagery extracted from Google Static Maps API using method in Sanya & Mwebaze (2018). It consists of land-use type buildings i.e. objects of interest in a satellite scene image is buildings or houses. The housing data was used as input to train the model, as proxy for indoor overcrowding - a known risk factor for our case study disease. All input data was pre-processed into feature vectors and normalized to (0,1) value range by min-max scaling or (-1,1) by standardizing (zero mean, unit variance).
Methods
We created disease density classes out of the epidemic data to make it a classification task. We did this by binning the normalized data such that each 250m2 grid assumed a class value depending on which bin its disease density lies in. Following the procedure in Robinson et al. (2017) for binning population data, we created a matrix C where an entry Ci = 0 if 5.74e− 05 ≤ di ≤ 1.52e− 4, 1 if 1.52e − 4 < di ≤ 2.46e − 4, and 2 if 2.46e − 4 < di ≤ 3.4e − 4, where di is normalized disease density. The classes correspond to the semantic labels 0, 1, 2 for high, low, and moderate disease density with 3,628, 6,472, and 1,970 membership count, respectively.
Let D be a grid of disease case counts for a region of interest at time t, C grid of disease density class label values, and S grid of housing concentration data. For every disease case count value di and density class label value ci there is an associated housing concentration image si.
We formulated the learning task as estimating unknown function f based on the idea of learning deep features over neighboring scene images (one neighbor in this case) using Eq. 11 (Figure 1a),
ci = { f(si, sj) if D(si, sj) < τ and arg minsi(D(si, sj)), f(si) if D(si, sj) > τ.
(11)
where ci is estimated disease density for a 250m2 patch of land on earth’s surface (approx. 224m2 pixels for an image), represented by image si. sj is image of a neighboring scene that is most similar to image of tile at location i, identified using method in Eq. 6.
We used CNN to estimate function f because the mapping from input data to disease incidence estimate is non-linear, noisy, and dependent on semantic content of input data. To train our model we used transfer learning by fine-tuning two pre-trained models on our data set i.e., Xception net and ResNet-50. Transfer learning eliminates network architecture design time while often achieving higher accuracy rates than training a model from scratch.
We applied the same regularization methods and other hyperparameter values (Table 4 in Appendix) to ensure uniformity across the proposed and baseline network architectures. Once a satisfactory model was achieved, it was trained on a combined data set (12,070 images) of training (9,656) and validation (2,414) before evaluating on the test set (1,189).

heading: 4.2 RESULTS

heading: 4.2.1 SIMILARITY METRIC LEARNING

text: Evaluation results for similarity metric learning for our proposed model built on top of ResNet-50 and Xception net are shown in Table 1. ResNet-50 performed better than Xception net on precision (0.39 vs. 0.70) and recall (0.10 vs. 0.60) on the task of detecting whether or not a pair of satellite scene images is similar, even though the overall performance is low for both pre-trained models (maximum OA score of 83 percent got with Xception net).

heading: 4.2.2 CLASSIFICATION PERFORMANCE

text:
Performance results for the baseline and proposed model are shown in Table 2. Overall, the Xception based model performed better than the ResNet-50 based one on the task of estimating disease density from housing satellite scene data when performance is measured using precision, recall, and F1-Score metrics. However, both models generally performed poorly, achieving a maximum overall accuracy score of 44 percent for ResNet-50. Our proposed model however, consistently out performed the baseline across both pre-trained models (scores marked with bold). For example, using Xception net as the Siamese network our model achieved precision score of 0.15 vs. 0.13, recall of 0.39 vs. 0.26, and f1-score of 0.22 vs. 0.18 for the baseline model, respectively. A similar performance trend is observed for the model based on ResNet-50.
Confusion matrix plots for baseline and our model built using Xception net are shown in Figure 3a and 3b, respectively (Appendix). On the other hand Figure 4 shows AUROC plots. Both show that our model performed better than the baseline on medium and low disease density classes with 58 vs. 48 percent and 47 vs. 37 percent, respectively. However, the model performed worse than the
baseline on high disease density class (0 vs. 13 percent, respectively). Generally, the results from both models are as good as chance.

heading: 4.2.3 REGRESSION PERFORMANCE

text: Table 3 gives MAE and MSE scores for baseline and proposed model. Overall, the model built using ResNet-50 performed better than the one built with Xception net when used to estimate disease density from housing concentration satellite scene image data. Again, our proposed model performed consistently (even though marginally) better than the baseline (scores marked with bold). It achieved MAE score of 0.35 vs. 0.38 and MSE score of 0.18 vs. 0.20 for the baseline, respectively. A similar trend is observed for the model built with Xception net as base.

heading: 5 DISCUSSION

text: Our model performed better than the baseline in both the classification and regression tasks for disease density estimation. For example, our model achieved 1 percentage point gain in accuracy over the baseline model. While a gain as result of deploying a siamese network to boost discriminative power of CNN for aerial scene image classification is consistent with findings in previous studies overall results from our model are poor. For instance, our model was only able to attain a maximum overall accuracy of 34 percent on the classification task. We would like to pin these poor results to a combination of three factors. First, the small data set used to train our model (12,070 images) could have impacted accuracy negatively, despite the use of regularization methods.It is therefore, possible that our model suffered the problem of overfitting. Secondly, our data set was unbalanced. It is likely that the extra parameter we introduced in the loss function to weight classes by giving higher importance to under-represented classes did not work as expected. The result could have been that our model did not learn all the necessary features required to make a prediction but rather could have resorted to guessing the output and hence, failing to generalize well over the test set. Class imbalance in our data set could also have negatively affected feature correlation which in turn could have reduced model performance. Besides, well-known methods for mitigating sample size bias in imbalanced data sets, for example by over-sampling under-represented classes Chawla et al. (2002) could not be applied directly to our data set without modifying the algorithm. That is because it was not immediately clear how to preserve spatial proximity between neighboring tiles, an idea that is central to learning deep features over neighboring scene images.However, despite the low overall performance by our model we have been able to demonstrate that it is possible to improve model accuracy by learning deep features over neighboring scene images in a disease density estimation task.

title: EXPLOITING SEMANTIC COHERENCE TO IMPROVE PREDICTION IN SATELLITE SCENE IMAGE ANALYSIS: APPLICATION TO DISEASE DENSITY ESTIMATION


summaries: This work proposes to leverage a pre-trained semantic segmentation network to learn semantically adaptive filters for self-supervised monocular depth estimation. Additionally, a simple two-stage training heuristic is proposed to improve depth estimation performance for dynamic objects that move in a way that induces small apparent motion and thus are projected to infinite depth values when used in an SfM-based supervision framework. Experimental results are shown on the KITTI benchmark, where the approach improves upon the state-of-the-art.

summaries: The paper proposes a using pixel-adaptive convolutions to leverage semantic labels in self-supervised monocular depth estimation. The semantic features are predicted by a pretrained network rather than relying on a ground truth. Moreover, a two-stage training process in proposed in order to filter out images leading to erroneous SfM predictions. The method is evaluated with different networks on the KITTY dataset.

abstractText: Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.†

authors:
Min Bai
Wenjie Luo
Kaustav Kundu
Raquel Urtasun

title: Exploiting semantic information and deep matching for optical flow

venue: In ECCV,

authors:
Holger Caesar
Varun Bankiti
Alex H. Lang
Sourabh Vora
Venice Erin Liong
Qiang Xu
Anush Krishnan
Yu Pan
Giancarlo Baldan
Oscar Beijbom

title: nuscenes: A multimodal dataset for autonomous driving

authors:
Vincent Casser
Soeren Pirk
Reza Mahjourian
Anelia Angelova

title: Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos

authors:
Po-Yi Chen
Alexander H. Liu
Yen-Cheng Liu
Yu-Chiang Frank Wang

title: Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation

venue: In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Djork-Arné Clevert
Thomas Unterthiner
Sepp Hochreiter

title: Fast and accurate deep network learning by exponential linear units (elus)

venue: In ICLR,

authors:
Marius Cordts
Mohamed Omran
Sebastian Ramos
Timo Rehfeld
Markus Enzweiler
Rodrigo Benenson
Uwe Franke
Stefan Roth
Bernt Schiele

title: The cityscapes dataset for semantic urban scene understanding

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
Jia Deng
Wei Dong
Richard Socher
Li jia Li
Kai Li
Li Fei-fei

title: Imagenet: A large-scale hierarchical image database

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
David Eigen
Christian Puhrsch
Rob Fergus

title: Depth map prediction from a single image using a multi-scale deep network

venue: In Advances in neural information processing systems,

authors:
Huan Fu
Mingming Gong
Chaohui Wang
Kayhan Batmanghelich
Dacheng Tao

title: Deep ordinal regression network for monocular depth estimation

authors:
Ravi Garg
Vijay Kumar BG
Gustavo Carneiro
Ian Reid

title: Unsupervised cnn for single view depth estimation: Geometry to the rescue

venue: In European Conference on Computer Vision,

authors:
Andreas Geiger
Philip Lenz
Christoph Stiller
Raquel Urtasun

title: Vision meets robotics: The kitti dataset

venue: The International Journal of Robotics Research,

authors:
Clément Godard
Oisin Mac Aodha
Gabriel J Brostow

title: Unsupervised monocular depth estimation with left-right consistency

venue: In CVPR,

authors:
Clément Godard
Oisin Mac Aodha
Michael Firman
Gabriel J. Brostow

title: Digging into selfsupervised monocular depth prediction

authors:
Vitor Guizilini
Sudeep Pillai
Rares Ambrus
Adrien Gaidon

title: Packnet-sfm: 3d packing for self-supervised monocular depth estimation

venue: arXiv preprint arXiv:1905.02693,

authors:
Fatma Güney
Andreas Geiger

title: Displets: Resolving stereo ambiguities using object knowledge

venue: In Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Xiaoyang Guo
Hongsheng Li
Shuai Yi
Jimmy Ren
Xiaogang Wang

title: Learning monocular depth by distilling cross-domain stereo networks

venue: In Proceedings of the European Conference on Computer Vision (ECCV),

title: Deep residual learning for image recognition

authors:
Max Jaderberg
Karen Simonyan
Andrew Zisserman

title: Spatial transformer networks. In Advances in neural information processing

authors:
Alex Kendall
Yarin Gal
Roberto Cipolla

title: Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

authors:
Alexander Kirillov
Ross Girshick
Kaiming He
Piotr Dollár

title: Panoptic feature pyramid networks

authors:
Maria Klodt
Andrea Vedaldi

title: Supervising the new with the old: Learning sfm from sfm

authors:
Jae-Han Lee
Minhyeok Heo
Kyung-Rae Kim
Chang-Su Kim

title: Single-image depth estimation based on fourier domain analysis

venue: In International Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Kuan-Hui Lee
German Ros
Jie Li
Adrien Gaidon

title: Spigan: Privileged adversarial learning from simulation

authors:
Bo Li
Chunhua Shen
Yuchao Dai
Anton Van
Mingyi He

title: Depth and surface normal estimation from monocular images using regression on deep features and hierarchical crfs

authors:
Jie Li
Allan Raventos
Arjun Bhargava
Takaaki Tagawa
Adrien Gaidon

title: Learning to fuse things and stuff

venue: arXiv preprint arXiv:1812.01192,

authors:
Zhizhong Li
Derek Hoiem

title: Learning without forgetting

venue: IEEE TPAMI,

authors:
Tsung-Yi Lin
Piotr Dollár
Ross Girshick
Kaiming He
Bharath Hariharan
Serge Belongie

title: Feature pyramid networks for object detection

authors:
Reza Mahjourian
Martin Wicke
Anelia Angelova

title: Unsupervised learning of depth and egomotion from monocular video using 3d geometric constraints

authors:
Fabian Manhardt
Wadim Kehl
Adrien Gaidon

title: Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape

authors:
Jeff Michels
Ashutosh Saxena
Andrew Y Ng

title: High speed obstacle avoidance using monocular vision and reinforcement learning

venue: In Proceedings of the 22nd international conference on Machine learning,

authors:
Matthias Ochs
Adrian Kretz
Rudolf Mester

title: Sdnet: Semantically guided depth estimation network

authors:
Adam Paszke
Sam Gross
Soumith Chintala
Gregory Chanan
Edward Yang
Zachary DeVito
Zeming Lin
Alban Desmaison
Luca Antiga
Adam Lerer

title: Automatic differentiation in pytorch

venue: NIPS-W,

authors:
Sudeep Pillai
Rares Ambrus
Adrien Gaidon

title: Superdepth: Self-supervised, super-resolved monocular depth estimation

venue: In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA),

authors:
Lorenzo Porzi
Samuel Rota Bulo
Aleksander Colovic
Peter Kontschieder

title: Seamless scene segmentation

authors:
Xiaojuan Qi
Renjie Liao
Zhengzhe Liu
Raquel Urtasun
Jiaya Jia

title: Geonet: Geometric neural network for joint depth and surface normal estimation

authors:
Hang Su
Varun Jampani
Deqing Sun
Orazio Gallo
Erik Learned-Miller
Jan Kautz

title: Pixeladaptive convolutional neural networks

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),

authors:
Benjamin Ummenhofer
Huizhong Zhou
Jonas Uhrig
Nikolaus Mayer
Eddy Ilg
Alexey Dosovitskiy
Thomas Brox

title: Demon: Depth and motion network for learning monocular stereo

venue: In IEEE Conference on computer vision and pattern recognition (CVPR),

authors:
Chaoyang Wang
José Miguel Buenaposada
Rui Zhu
Simon Lucey

title: Learning depth from monocular videos using direct methods

authors:
Zhou Wang
Alan C Bovik
Hamid R Sheikh
Eero P Simoncelli

title: Image quality assessment: from error visibility to structural similarity

venue: IEEE transactions on image processing,

authors:
Yuwen Xiong
Renjie Liao
Hengshuang Zhao
Rui Hu
Min Bai
Ersin Yumer
Raquel Urtasun

title: Upsnet: A unified panoptic segmentation network

authors:
Nan Yang
Rui Wang
Jörg Stückler
Daniel Cremers

title: Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry

authors:
Zhenheng Yang
Peng Wang
Wei Xu
Liang Zhao
Ramakant Nevatia

title: Unsupervised learning of geometry with edge-aware depth-normal consistency

venue: arXiv preprint arXiv:1711.03665,

authors:
Zhichao Yin
Jianping Shi

title: Geonet: Unsupervised learning of dense depth, optical flow and camera pose

authors:
Huangying Zhan
Ravi Garg
Chamara Saroj Weerasekera
Kejie Li
Harsh Agarwal
Ian Reid

title: Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction

authors:
Tinghui Zhou
Matthew Brown
Noah Snavely
David G Lowe

title: Unsupervised learning of depth and ego-motion from video

authors:
Yuliang Zou
Zelun Luo
Jia-Bin Huang

title: Df-net: Unsupervised joint learning of depth and flow using cross-task consistency

venue: In European Conference on Computer Vision, 2018

heading: 1 INTRODUCTION

text:
Accurate depth estimation is a key problem in computer vision and robotics, as it is instrumental for perception, navigation, and planning. Although perceiving depth typically requires dedicated sensors (e.g., stereo rigs, LiDAR), learning to predict depth from monocular imagery can provide useful cues for a wide array of tasks (Michels et al., 2005; Kendall et al., 2018; Manhardt et al., 2019; Lee et al., 2019). Going beyond supervised learning from direct measurements (Eigen et al., 2014), self-supervised methods exploit geometry as supervision (Guo et al., 2018; Pillai et al., 2019; Zou et al., 2018; Yang et al., 2017), therefore having the potential to leverage large scale datasets of raw videos to outperform supervised methods (Guizilini et al., 2019).
Although depth from a single image is an ill-posed inverse problem, monocular depth networks are able to make accurate predictions by learning representations connecting the appearance of scenes and objects with their geometry in Euclidean 3D space. Due to perspective, there is indeed an equivariance relationship between the visual appearance of an object in 2D and its depth, when conditioned on the object’s category. For instance, a car 25 meters away appears smaller (on the image plane) than a car only 5 meters away but bigger than a truck 50 meters away. Current depth estimation methods either do not leverage this structure explicitly or rely on strong semantic supervision to jointly optimize geometric consistency and a semantic proxy task in a multi-task objective (Ochs et al., 2019; Chen et al., 2019), thus departing from the self-supervised paradigm.
In this paper, we explore how we can leverage semantic information to improve monocular depth prediction in a self-supervised way. Our main contribution is a novel architecture that uses a fixed pre-trained semantic segmentation network to guide geometric representation learning in a selfsupervised monocular depth network. In contrast to standard convolutional layers, our architecture uses pixel-adaptive convolutions (Su et al., 2019) to learn semantic-dependent representations that can better capture the aforementioned equivariance property. Leveraging semantics may nonetheless introduce category-specific biases. Our second contribution is a two-stage training process where we automatically detect the presence of a common bias on dynamic objects (projections at infinity) and resample the training set to de-bias it. Our method improves upon the state of the art in selfsupervised monocular depth estimation on the standard KITTI benchmark (Geiger et al., 2013), both on average over pixels, over classes, and for dynamic categories in particular.
†Source code and pretrained models are available on https://github.com/TRI-ML/packnet-sfm

heading: 2 RELATED WORK

text:
Since the seminal work of Eigen et al. (2014), substantial progress has been done to improve the accuracy of supervised depth estimation from monocular images, including the use of Conditional Random Fields (CRFs) (Li et al., 2015), joint optimization of surface normals (Qi et al., 2018), fusion of multiple depth maps (Lee et al., 2018), and ordinal classification (Fu et al., 2018). Consequently, as supervised techniques for depth estimation advanced rapidly, the availability of largescale depth labels became a bottleneck, especially for outdoor applications. Garg et al. (2016) and Godard et al. (2017) provided an alternative self-supervised strategy involving stereo cameras, where Spatial Transformer Networks (Jaderberg et al., 2015) can be used to geometrically warp, in a differentiable way, the right image into a synthesized left image, using the predicted depth from the left image. The photometric consistency loss between the resulting synthesized and original left images can then be minimized in an end-to-end manner using a Structural Similarity term (Wang et al., 2004) and additional depth regularization terms. Following Godard et al. (2017) and Ummenhofer et al. (2017), Zhou et al. (2017) generalized this to the purely monocular setting, where a depth and a pose networks are simultaneously learned from unlabeled monocular videos. Rapid progress in terms of architectures and objective functions (Yin & Shi, 2018; Mahjourian et al., 2018; Casser et al., 2019; Zou et al., 2018; Klodt & Vedaldi, 2018; Wang et al., 2018; Yang et al., 2018) have since then turned monocular depth estimation into one of the most successful applications of self-supervised learning, even outperforming supervised methods (Guizilini et al., 2019).
The introduction of semantic information to improve depth estimates has been explored in prior works, and can be broadly divided into two categories. The first one uses semantic (or instance) information to mask out or properly model dynamic portions of the image, which are not accounted for in the photometric loss calculation. Güney & Geiger (2015) leveraged object knowledge in a Markov Random Field (MRF) to resolve stereo ambiguities, while Bai et al. (2016) used a conjunction of instance-level segmentation and epipolar constraints to reduce uncertainty in optical flow estimation. Casser et al. (2019) used instance-level masks to estimate motion models for different objects in the environment, and account for their external motion in the resulting warped image. The second category attempts to learn both tasks in a single framework, and uses consistency losses to ensure that both are optimized simultaneously and regularize each other, so the information contained in one task can be transferred to improve the other. For instance, Ochs et al. (2019) estimated depth with an ordinal classification loss similar to the standard semantic classification loss, and used empirical weighting to combine them into a single loss for optimization. Similarly, Chen et al. (2019) used a unified conditional decoder that can generate either semantic or depth estimates, and both outputs are used to generate a series of losses also combined using empirical weighting to generate the final loss to be optimized.
Our approach focuses instead on representation learning, exploiting semantic features into the selfsupervised depth network by using a pretrained semantic segmentation network to guide the generation of depth features. This is done using pixel-adaptive convolutions, recently proposed in Su et al. (2019) and applied to tasks such as depth upsampling using RGB images for feature guidance. We show that different depth networks can be readily modified to leverage this semantic feature guidance, ranging from widely used ResNets (He et al., 2016) to the current state-of-the-art PackNet (Guizilini et al., 2019), with a consistent gain in performance across these architectures.

heading: 3 SELF-SUPERVISED STRUCTURE-FROM-MOTION

text: Our semantically-guided architecture is developed within a self-supervised monocular depth estimation setting, commonly known as structure-from-motion (SfM). Learning in a self-supervised structure-from-motion setting requires two networks: a monocular depth model fD : I → D, that outputs a depth prediction D̂ = fD(I(p)) for every pixel p in the target image I; and a monocular ego-motion estimator fx : (It, IS) → xt→S , that predicts the 6 DoF transformations for all s ∈ S given by xt→s = ( R t0 1 ) ∈ SE(3) between the target image It and a set of temporal context source images Is ∈ IS . In all reported experiments we use It−1 and It+1 as source images.

heading: 3.1 THE SELF-SUPERVISED OBJECTIVE LOSS

text:
We train the depth and pose networks simultaneously, using the same protocols and losses as described in Guizilini et al. (2019). Our self-supervised objective loss consists of an appearance matching term Lp that is imposed between the synthesized Ît and original It target images, and a depth regularization term Ls that ensures edge-aware smoothing in the depth estimates D̂t. The final objective loss is averaged per pixel, pyramid scale and image batch, and is defined as:
L(It, Ît) = Lp(It, Ît) + λ1 Ls(D̂t) (1) where λ1 is a weighting coefficient between the photometricLp and depth smoothnessLs loss terms. Following Godard et al. (2017) and Zhou et al. (2017), the similarity between synthesized Ît and original It target images is estimated using a Structural Similarity (SSIM) term (Wang et al., 2004) combined with an L1 loss term, inducing the following overall photometric loss:
Lp(It, Ît) = α 1− SSIM(It, Ît)
2 + (1− α) ‖It − Ît‖ (2)
In order to regularize the depth in low gradient regions, we incorporate an edge-aware term similar to Godard et al. (2017). This loss is weighted for each of the pyramid levels, decaying by a factor of 2 on each downsampling, starting with a weight of 1 for the 0th pyramid level.
Ls(D̂t) = |δxD̂t|e−|δxIt| + |δyD̂t|e−|δyIt| (3) We also incorporate some of the insights introduced in Godard et al. (2018), namely auto-masking, minimum reprojection error, and inverse depth map upsampling to further improve depth estimation performance in our self-supervised monocular setting.

heading: 3.2 DEPTH AND POSE NETWORKS

text: Our baseline depth and pose networks are based on the PackNet architecture introduced by Guizilini et al. (2019), which proposes novel packing and unpacking blocks to respectively downsample and upsample feature maps during the encoding and decoding stages. This network was selected due to its state-of-the-art performance in the task of self-supervised monocular depth estimation, so we can analyze if our proposed architecture is capable of further improving the current state-of-the-art. However, there are no restrictions as to which models our proposed semantically-guided architecture can be applied to, and in Section 5.4 we study its application to different depth networks.

heading: 4 SEMANTICALLY-GUIDED GEOMETRIC REPRESENTATION LEARNING

text: In this section, we describe our method to inject semantic information into a self-supervised depth network via its augmentation with semantic-aware convolutions. Our proposed architecture is depicted in Figure 2 and is composed of two networks: a primary one, responsible for the generation of depth predictions D̂ = fD(I(p)); and a secondary one, capable of producing semantic predictions. Only the first network is optimized during self-supervised learning; the semantic network is initialized from pretrained weights and is not further optimized. This is in contrast to the common practice of supervised (ImageNet) pretraining of depth encoders (Godard et al., 2018; Casser et al., 2019; Zou et al., 2018): here instead of fine-tuning from pre-trained weights, we preserve these secondary weights to guide the feature learning process of the primary depth network. Our approach also differs from learning without forgetting (Li & Hoiem, 2017) by leveraging fixed intermediate feature representations as a way to maintain consistent semantic guidance throughout training.

heading: 4.1 SEMANTICALLY-GUIDED DEPTH FEATURES

text:
We leverage the information from the pretrained semantic network in the depth network through the use of pixel-adaptive convolutions (Su et al., 2019). They were recently proposed to address some limitations inherent to the standard convolution operation, namely its translation invariance making it content-agnostic. While this significantly reduces the number of parameters of the resulting network, this might also lead to sub-optimal solutions under certain conditions important for geometric representation learning. For example, spatially-shared filters globally average the loss gradients over the entire image, forcing the network to learn weights that cannot leverage location-specific information beyond their limited receptive fields. Content-agnostic filters are unable to distinguish between different pixels that are visually similar (i.e. dark areas due to shadows or black objects) or generalize to similar objects that are visually different (i.e. cars with varying colors). In this work, we use pixel-adaptive convolutions to produce semantic-aware depth features, where the fixed information encoded in the semantic network is used to disambiguate geometric representations for the generation of multi-level depth features.
As shown in Figure 2, we extract multi-level feature maps from the semantic network. For each feature map, we apply a 3 × 3 and a 1 × 1 convolutional layer followed by Group Normalization (Wu & He, 2018) and ELU non-linearities (Clevert et al., 2016). These processed semantic feature maps are then used as guidance on their respective pixel-adaptive convolutional layers, following the formulation proposed in Su et al. (2019):
v′i = ∑ j∈Ω(i) K(fi, fj)W[pi − pj ]vj + b (4)
In the above equation, f ∈ RD are processed features from the semantic network that will serve to guide the pixel-adaptive convolutions from the depth network, p = (x, y)T are pixel coordinates, with [pi − pj ] denoting 2D spatial offsets between pixels, Wk×k are convolutional weights with kernel size k, Ωi defines a k×k convolutional window around i, v is the input signal to be convolved, and b ∈ R1 is a bias term. K is the kernel used to calculate the correlation between guiding features, here chosen to be the standard Gaussian kernel:
K(fi, fj) = exp ( −1
2 (fi − fj)TΣ−1ij (fi − fj)
) (5)
where Σij is the covariance matrix between features fi and fj , here chosen to be a diagonal matrix σ2 ·ID, with σ as an extra learnable parameter for each convolutional filter. These kernel evaluations can be seen as a secondary set of weights applied to the standard convolutional weights, changing their impact on the resulting depth features depending on the content stored in the guiding semantic features. For example, the information contained in depth features pertaining to the sky should not be used to generate depth features describing a pedestrian, and this behavior is now captured as a larger distance between their corresponding semantic features, which in turn produces smaller weights for that particular convolutional filter. Note that the standard convolution can be considered a special case of the pixel-adaptive convolution, where ∀ ij,K(fi, fj) = 1.

heading: 4.2 SEMANTIC GUIDANCE NETWORK

text: As the secondary network used to provide semantic guidance for the generation of depth features, we use a Feature Pyramid Network (FPN) with ResNet backbone (Lin et al., 2017). This architecture has been shown to be efficient for both semantic and instance-level predictions towards panoptic segmentation (Kirillov et al., 2019; Li et al., 2018; Xiong et al., 2019; Porzi et al., 2019). While our proposed semantically-guided architecture is not restricted to any particular network, we chose this particular implementation to facilitate the future exploration of different sources for guidance information. Architectural details follow the protocols described in Li et al. (2018), and unless mentioned otherwise the same pretrained model was used in all reported experiments. The semantic network is assumed fixed, pretrained on a held out dataset different than the raw data used for selfsupervised learning, i.e. we do not require any semantic ground truth on the target dataset.

heading: 4.3 TWO-STAGE TRAINING

text:
One well-known limitation of the self-supervised photometric loss is its inability to model dynamic objects, due to a static world assumption that only accounts for camera ego-motion (Godard et al., 2018; Casser et al., 2019). A resulting common failure mode is the infinite depth problem, which is caused by the presence of objects moving at the same speed as the camera. This typically causes distinct holes in the predicted depth maps, with arbitrarily large values where these objects should be. This severely hinders the applicability of such models in real-world applications, particularly for automated driving, where the ability to detect and properly model dynamic objects is crucial. Moreover, this limitation may be further accentuated in our proposed semantically-guided architecture, as the infinite depth problem occurs mostly on dynamic categories (i.e. cars and motorcycles) and the semantic-aware features may reinforce this bias.
We propose a simple and efficient two-stage training method to detect and remove this bias from the training set. In the first stage, we learn a standard depth network on all available training data. This network, exhibiting the infinite depth problem, is then used to resample the dataset by automatically filtering out sequences with infinite depth predictions that violate a basic geometric prior. We indeed find that depth predictions for pixels corresponding to the nearby ground plane are generally robust. This enables getting a coarse estimate of the ground plane using RANSAC and detecting the number of pixels whose predicted depth projects them significantly below the ground. If that number is above a threshold, then the corresponding image is subsequently ignored (we found a conservative threshold of 10 to work well in all our experiments, filtering out roughly 5% of the KITTI training dataset). During the second stage, we retrain the network on the subsampled dataset (from scratch to avoid the previous local optimum). As this subsampled dataset is de-biased, the network learns better depth estimates on dynamic objects. This process can be repeated, but we find that two stages are enough to remove any traces of infinite depth in our experiments, as shown in Figure 3.

heading: 5 EXPERIMENTAL RESULTS

heading: 5.1 DATASETS

text:
We use the standard KITTI benchmark (Geiger et al., 2013) for self-supervised training and evaluation. More specifically, we adopt the training, validation and test splits used in Eigen et al. (2014) with the pre-processing from Zhou et al. (2017) to remove static frames, which is more suitable for
monocular self-supervised learning. This results in 39810 images for training, 4424 for validation, and 697 for evaluation. Following common practice, we pretrain our depth and pose networks on the CityScapes dataset (Cordts et al., 2016), consisting of 88250 unlabeled images. Unless noted otherwise, input images are downsampled to 640 x 192 resolution and output inverse depth maps are upsampled to full resolution using bilinear interpolation. Our fixed semantic segmentation network is pretrained on Cityscapes, achieving a mIoU of 75% on the validation set.

heading: 5.2 IMPLEMENTATION DETAILS

text:
We implement our models with PyTorch (Paszke et al., 2017) and follow the same training protocols of Guizilini et al. (2019) when optimizing our depth and pose networks. The initial training stage is conducted on the CityScapes dataset for 50 epochs, with a batch size of 4 per GPU and initial depth and pose learning rates of 2 · 10−4 and 5 · 10−4 respectively, that are halved every 20 epochs. Afterwards, the depth and pose networks are fine-tuned on KITTI for 30 epochs, with the same parameters and halving the learning rates after every 12 epochs. This fine-tuning stage includes the proposed architecture, where information from the fixed semantic network, pretrained separately, is used to directly guide the generation of depth features. There is no direct supervision at any stage during depth training, all semantic information is derived from the fixed secondary network.
When pretraining the semantic segmentation network, we use a ResNet-50 backbone with Imagenet (Deng et al., 2009) pretrained weights and optimize the network for 48k iterations on the CityScapes dataset with a learning rate of 0.01, momentum of 0.9, weight decay of 10−4, and a batch size of 1 per GPU. Random scaling between (0.7, 1.3), random horizontal flipping, and a crop size of 1000×2000 are used for data augmentation. We decay the learning rate by a factor of 10 at iterations 36k and 44k. Once training is complete, the semantic segmentation network is fixed and becomes the only source of semantic information when fine-tuning the depth and pose networks on KITTI.

heading: 5.3 DEPTH ESTIMATION PERFORMANCE

text:
Our depth estimation results are summarized in Table 1, where we compare our proposed architecture with other published works. From these results we can see that the introduction of semanticallyguided geometric representation learning further improves upon the current state of the art in selfsupervised monocular depth estimation from Guizilini et al. (2019), which served as our baseline. Our approach also outperforms other methods that leverage semantic information by a substantial margin, even those using ground-truth KITTI semantic segmentation and depth labels during train-
ing (Ochs et al., 2019). Furthermore, in Figure 5 we also present qualitative results showing the improvements in depth estimation generated by our proposed framework, compared to our baseline. Note how our semantically-guided architecture produces sharper boundaries and better object delineation, especially in structures further away or not clearly distinguishable in the input image.

heading: 5.4 ABLATIVE ANALYSIS

heading: 5.4.1 DIFFERENT DEPTH NETWORKS

text: To better evaluate our main contribution, we provide an ablative analysis showing how it generalizes to different depth networks. To this end, we consider two variations of the widely used ResNet architecture as the encoder for our depth network: ResNet-18 and ResNet-50 (the same pretrained semantic network was used in all experiments). Depth estimation results considering these variations are shown in Table 2, where we can see that our proposed semantically-guided architecture is able to consistently improve the performance of different depth networks, for all considered metrics.

heading: 5.4.2 CLASS-SPECIFIC DEPTH PERFORMANCE

text:
To further showcase the benefits of our semantically-guided architecture, we also provide classspecific evaluation metrics, as shown in Figure 4. As we do not have ground-truth semantic segmentation for these images, we use the prediction of the semantic network to bin pixels per predicted category, and evaluate only on those pixels. From these results we can see that our proposed architecture consistently improves depth performance for pixels across all predicted classes, especially those containing fine-grained structures and sharp boundaries, e.g. poles and traffic signs.
We also measure the impact of our two-stage training process, which is expected to address the infinite depth problem in dynamic objects. Although we find the pixel-average difference in performance to not be significant (see Table 2), there is a significant improvement in class-average depth estimation, from 0.121 to 0.117 Abs-Rel. This is because the number of pixels affected by the infinite depth problem is vastly smaller than the total number of pixels. However, when considering class-average depth evaluation, the improvement over classes such as cars (0.200 to 0.177 Abs-Rel) and motorcycles (0.091 to 0.069) becomes statistically significant. This further exemplifies the importance of fine-grained metrics in depth evaluation, so these underlying behaviors can be properly observed and accounted for in the development of new techniques.

heading: 6 CONCLUSION

text: This paper introduces a novel architecture for self-supervised monocular depth estimation that leverages semantic information from a fixed pretrained network to guide the generation of multi-level depth features via pixel-adaptive convolutions. Our monodepth network learns semantic-aware geometric representations that can disambiguate photometric ambiguities in a self-supervised learning structure-from-motion context. Furthermore, we introduce a two-stage training process that resamples training data to overcome a common bias on dynamic objects resulting in predicting them at infinite depths. Our experiments on challenging real-world data shows that our proposed architecture consistently improves the performance of different monodepth architectures, thus establishing a new state of the art in self-supervised monocular depth estimation. Future directions of research include leveraging other sources of guidance (i.e. instance masks, optical flow, surface normals), as well as avenues for self-supervised fine-tuning of the semantic network.

heading: A PRE-TRAINING THE SEMANTIC SEGMENTATION NETWORK

text:
The introduction of a semantic segmentation network to the depth estimation task increases the depth estimation performance, however it also increases model complexity (e.g. number of trainable parameters). To investigate that the increased performance for the depth estimation task is indeed due to the semantic features encoded in the secondary network, we perform an in-depth analysis (summarized in Table 3) where we explore the impact of pre-training the semantic segmentation network before it is used to guide the generation of depth features. From these results we can see that the presence of semantic information encoded in the secondary network indeed leads to an increase in performance, and that fine-tuning this secondary network for the speficic task of depth estimation actually decreases performance.
In the first two rows an untrained semantic network is utilized, with only its encoder initialized from ImageNet (Deng et al., 2009) weights. Two different scenarios are explored: in the first one (D) only the depth network is fine-tuned in a self-supervised fashion, while in D+S both networks are fine-tuned together in the same way. As expected, using untrained features as guidance leads to significantly worse results, since there is no structure encoded in the secondary network and the primary network needs to learn to filter out all this spurious information. When both networks are fine-tuned simultaneously, results improve because now the added complexity from the secondary network can be leveraged for the task of depth estimation, however there is still no improvement over the baseline.
Next, the semantic network was pre-trained on only half of the CityScapes (Cordts et al., 2016) dataset (samples chosen randomly), leading to a worse semantic segmentation performance (validation mIoU of around 70% vs. 75% for the fully trained one). This partial pre-training stage was enough to enable the transfer of useful information between networks, leading to improvements over the baseline. Interestingly, fine-tuning both networks for the task of depth estimation actually hurt performance this time, which we attribute to forgetting the information contained in the secondary network, as both networks are optimized for the depth task. When the semantic network is pretrained with all of CityScapes (last two rows), these effects are magnified, with fine-tuning only the depth network leading to our best reported performance (Table 1) and fine-tuning both networks again leading to results similar to the baseline.

heading: B UNCERTAINTY AND GENERALIZATION TO DIFFERENT OBJECTS

text:
In a self-supervised setting, increasing the number of unlabeled videos used for depth training is expected to lead to an increasing specialization away from the domain in which the semantic network was pre-trained. This might result in harmful guidance if our method is not robust to this gap. However, our approach does not use semantic predictions directly, but rather the decoded features of the semantic network themselves, which represent general appearance information that should be more robust to this domain gap. To validate our hypothesis, we further explore the impact
of erroneous semantic information in the performance of our proposed semantically-guided depth framework. In Figure 6 we present qualitative results highlighting situations in which our pretrained semantic network failed to generate correct semantic predictions for certain objects in the scene, and yet our proposed framework was still able to properly recover depth values for that portion of the environment. These exemplify possible scenarios for erroneous semantic prediction.
• Imprecise boundaries: in the first row, we can see that the semantic segmentation network does not correctly detect the traffic sign, yet the semantically-guided depth network predicts its shape and depth accurately.
• Wrong classification: in the second row, the truck was mistakenly classified as partially “road” and “building”, however our semantically-guided depth network was still able to properly recover its overall shape with sharp delineation that was not available from its semantic contour. A similar scenario happens in the same image, with “fence” being partially labeled as “bicycle”.
• Missing ontology: there is no “trash can” class on the CityScapes ontology, however in the third row our semantically-guided depth network was able to correctly reconstruct such object even though it was classified as “fence”, similarly to its surroundings.
• Object Hallucination: in the fourth row, the contour of a “person” was erroneously introduced in the image and correctly removed by our semantically-guided framework.
These examples are evidence that our proposed framework is able to reason over the uncertainty inherent to semantic classification, leveraging this information when accurate to achieve the results reported in this paper, but also discarding it if necessary to generate a better reconstruction according to the self-supervised photometric loss.

heading: C GENERALIZATION TO DIFFERENT DATASETS

text:
In the previous sections, we show that our proposed framework is robust to a degraded semantic network, both by pretraining the semantic network with fewer annotated labels (Appendix A) and also by providing evidence that the depth network is able to reason over erroneous predictions to still generate accurate reconstructions (Appendix B). We now go one step further and analyze how our proposed semantically-guided framework generalizes to a dataset that was used neither during pre-training nor for fine-tuning. To this end, we evaluate our KITTI depth model on the recently released NuScenes dataset (Caesar et al., 2019). The official NuScenes validation split is used, containing 6019 images from the front camera with ground-truth depth maps generated by LiDAR reprojection. Results presented in Table 4 provide additional evidence that our method indeed results in generalization improvements, even on significantly different data from different platforms
and environments (Karlsruhe, Germany for KITTI vs Boston, USA and Singapore for NuScenes), outperforming the state of the art methods and our baseline (Guizilini et al., 2019).

title: ING FOR SELF-SUPERVISED MONOCULAR DEPTH