Reference-based Image Super-Resolution with Deformable Attention Transformer

Reference-based image super-resolution (RefSR) aims to exploit auxiliary reference (Ref) images to super-resolve low-resolution (LR) images. Recently, RefSR has been attracting great attention as it provides an alternative way to surpass single image SR. However, addressing the RefSR problem has two critical challenges: (i) It is difficult to match the correspondence between LR and Ref images when they are significantly different; (ii) How to transfer the relevant texture from Ref images to compensate the details for LR images is very challenging. To address these issues of RefSR, this paper proposes a deformable attention Transformer, namely DATSR, with multiple scales, each of which consists of a texture feature encoder (TFE) module, a reference-based deformable attention (RDA) module and a residual feature aggregation (RFA) module. Specifically, TFE first extracts image transformation (e.g., brightness) insensitive features for LR and Ref images, RDA then can exploit multiple relevant textures to compensate more information for LR features, and RFA lastly aggregates LR features and relevant textures to get a more visually pleasant result. Extensive experiments demonstrate that our DATSR achieves state-of-the-art performance on benchmark datasets quantitatively and qualitatively.

Introduction
Single image super-resolution (SISR), which aims at recovering a high-resolution (HR) image from a low-resolution (LR) input, is an active research topic due to its high practical values [13,51,18,49,41,15,46,21,16,14,9,20]. However, SISR is a highly ill-posed problem since there exist multiple HR images that can degrade to the same LR image [38,8]. While real LR images usually have no corresponding HR ground-truth (GT) images, one can easily find a high-quality image as a reference (Ref) image with high-frequency details from various sources, such  [12]. When the brightness of LR and Ref image is different, our method performs better than C 2 -Matching [12] in transferring relevant textures from the Ref image to the SR image, which is closer to the ground-truth image.
as photo albums, video frames, and web image search, which has similar semantic information (such as content and texture) to the LR image. Such an alternative SISR method is referred to as reference-based super-resolution (RefSR), which aims to transfer HR textures from the Ref images to super-resolved images and has shown promising results over SISR. Although various RefSR methods [12,27,47,45] have been recently proposed, two challenges remain unsolved for SR performance improvement.
First, it is difficult to match the correspondence between the LR and Ref images especially when their distributions are different. For example, the brightness of the Ref images is different from that of the LR images. Existing methods [56,48] mostly match the correspondence by estimating the pixel or patch similarity of texture features between LR and Ref images. However, such similarity metric is sensitive to image transformations, such as brightness and color of images. Recently, the state-of-the-art (SOTA) method C 2 -Matching [12] trains a feature extractor, which demonstrates strong robustness to scale and rotation. However, it neglects to explore the effects of brightness, contrast, and color of images. As a result, this method may transfer inaccurate textures from the Ref image, when the Ref images have different brightness from the LR image, as shown in Fig. 1. Based on the observation and analyses, we can see that the quality of correspondence is affected by the similarity metric and the distribution gap between the LR and Ref images.
On the other hand, some methods [57,34] adopt optical flow or deformable convolutions [4,59,3,42] to align spatial features between the Ref and LR images. However, these methods may find an inaccurate correspondence when the distance between the LR and Ref images is relatively large. With the inaccurate correspondence, their performance would deteriorate seriously since the irrelevant texture cannot provide meaningful details. Therefore, how to accurately match the correspondence between the Ref and LR images is a challenging problem as it affects the quality of super-resolved results.
Second, it is also challenging to transfer textures of the high-quality Ref images to restore the HR images. One representative work CrossNet [57] estimates the flow from the Ref image to the LR image and then warp the features based on the optical flow. However, the optical flow may be inaccurate, since the Ref and LR images could be significantly different. In addition, most existing methods [56,48,27] search the most similar textures and the corresponding position, and then swap the texture features from the Ref image. As a result, these methods may transfer irrelevant textures to the output and have poor SR performance, when the original estimated flow or position is inaccurate. Hence, it is important and necessary to explore a new architecture to adaptively transfer texture and mitigate the impact of inaccurate correspondence in the Ref image.
To address the above two challenges, we propose a novel deformable attention Transformer, namely DATSR, for reference-based image super-resolution. DATSR is built on the U-Net and consists of three basic modules, including texture feature encoders, deformable attention, and residual feature aggregation. Specifically, we first use texture feature encoders to extract multi-scale features with different image transformations. Then, we propose a reference-based deformable attention to discover the multiple relevant correspondences and adaptively transfer the textures. Last, we fuse features and reconstruct the SR images using residual feature aggregation. We conduct extensive comparisons with recent representative SOTA methods on benchmark datasets. The quantitative and visual results demonstrate that our DATSR achieves the SOTA performance.
The main contributions are summarized as follows: -We propose a novel reference-based image super-resolution with deformable attention transformer (DATSR), which is end-to-end trainable by incorporating Transformer into RefSR. Compared with existing RefSR methods, our DATSR performs more robust correspondence matching and texture transfer and subsequently achieves SOTA performance quantitatively and visually. -

Related Work
We will briefly introduce two related super-resolution paradigms, including single image super-resolution and reference-based image super-resolution.
Single image super-resolution (SISR). The goal of SISR is to recover highresolution (HR) images from the low-resolution (LR) images. Recent years have witnessed significant achievements of using deep neural networks to solve SISR [6,55]. SRCNN [6] is the pioneer work of exploiting deep convolutional networks to map LR image into HR image. To further improve SR performance, researchers resort to employing deeper neural networks with attention mechanisms and residual blocks [22,33,55,19,31,20,54,23,21,50,36,5]. However, it is difficult for traditional SISR methods to produce realistic images when the HR textures are highly degraded. To relieve this, some SR methods [17,44,53,11,40,43,58] adopt generative adversarial networks (GANs) to further improve the perceptual quality of the super-resolved outputs.
Reference-based image super-resolution (RefSR  [34] proposes a similarity search and extraction network and it is aware of the best matching position and the relevancy of the best match. To improve the performance, TTSR [48] proposes a hard and soft attention for texture transfer and synthesis. Instead of using the features of a classifier, E2ENT 2 [45] transfers texture features by using a SR task-specific features. To improve the efficiency of matching, MASA [27] proposes a coarse-to-fine correspondence matching module and a spatial adaptation module to map the distribution of the Ref features to that of the LR features. Recently, a strong RefSR method C 2 -Matching [12] first proposes a contrastive correspondence network to learn correspondence, and then adopts a teacher-student correlation distillation to improve LR-HR matching, and last uses a residual feature aggregation to synthesize HR images. It should be noted that RefSR can be extended to the case of multiple reference images, called Multi-RefSR, which aims to transfer the texture features from multiple Ref images to the SR image. Recently, a content independent multi-reference super-resolution model CIMR-SR [47] is proposed to transfer the HR textures from multiple reference images. To improve the performance, AMRSR [32] proposes an attention-based multi-reference super-resolution network to match the most similar textures from multiple reference images. Different from RefSR, Multi-RefSR can exploit more training information as it has multiple Ref images. In this paper, we mainly study RefSR and train the model with single Ref image. Nevertheless, we still compare our model with the above Multi-RefSR methods to further demonstrate the effectiveness of our DATSR.

Proposed Method
Due to the the intrinsic complexity of RefSR, we divide the problem into two main sub-tasks: correspondence matching and texture transfer. To address these, we propose a multi-scale reference-based image SR with deformable Transformer, as shown in Fig. 2 where E q l , E k l and E v l are feature encoders at the l-th scale. In our architecture, we use three scales in the texture feature encoders. With the help of the multiscale features in U-Net, we are able to alleviate the resolution gap between the Ref and LR images since they contain the complementary scale information.
For the distribution gap, we augment images with different image transformations (e.g., brightness, contrast and hue) in the training to improve the robustness of our model. In addition to data augmentation, we use contrastive learning to train the encoder be less sensitive to different image transformations, inspired by [12]. To estimate the stable correspondence between X LR↑ and X Ref , the feature encoders E q l and E k l are the same, and the feature encoder E r l is pretrained and fixed in the training. In contrast, TTSR [48] directly uses a learnable feature encoder, resulting in limited performance since the textures are changing during training and the correspondence matching is unstable. For C 2 -Matching [12], it neglects to improve the robustness to brightness, contrast and hue. To address these, we propose to learn robust multi-scale features Q l , K l , V l , which can be regraded as Query, Key, and Value, and can be used in our attention mechanism conditioned on the LR features.

Reference-based Deformable Attention
Existing attention-based RefSR methods (e.g., [48]) tend to suffer from limited performance when the most relevant features between LR and Ref images are inaccurate, i.e., the learned LR features may not well match the Ref features. To address this, we propose a new reference-based attention mechanism, called RefAttention, as shown in Fig. 3. Formally, given Query Q l , Key K l , Value V l , and LR features F l , the attention feature A l is defined as follows: Different from existing attention mechanism [39], our attention is conditioned on the LR features and designed for the RefSR task. In Fig. 3, we denoted by A l and F l in the downscaling process, andÃ l andF l in the upscaling process. σ(·) is a correspondence matching function to calculate the relevance between the Ref and LR images. Based on the relevance, we propose a texture transfer function T (·) to transfer the textures from the Ref to the LR image.
Correspondence matching. The first important sub-task in RefSR is to match correspondences between LR and Ref images. Most existing methods [56,48] are sensitive to different image transformations (e.g., brightness, contrast and hue) and may match inaccurate correspondences. To relieve this issue, we propose a correspondence matching module in our RefAttention, as shown in Fig. 3. Specifically, we estimate the relevance between X LR↑ and X Ref by calculating similarity between Q l ∈ R C×H1×W1 and K l ∈ R C×H2×W2 . First, we unfold Q l and K l into patches Then, for the given query q i in Q ′ , the top K relevant positions in K ′ can be calculated by normalized inner product, where q i = q i /∥q i ∥ and k j = k j /∥k j ∥ are normalized features, and TopK(·) is a function and returns top K relevant positions the i-th element of P l , and the position p 1 i is the most relevant position in the Ref image to the i-th position in LR. When K > 1, it helps discover multiple correspondences, motivated by KNN [24]. For fair comparisons with other RefSR methods, we set K = 1 and exploit the most relevant position in the experiments.
Similarity-aware texture transfer. The second important sub-task in RefSR is to transfer textures from Ref images to LR images based on the matched correspondence. Most existing RefSR methods [56,48] directly swap the most relevant texture from Ref image. However, it may degrade the performance when the most relevant texture is inaccurate. To address this, we propose to improve the deformable convolution (DCN) [4,59] to transfer the texture around every position p k i of Ref images. Specifically, let ∆p k i be the spatial difference between the position p i and the k-th relevant position p k i , i.e., ∆p k i = p k i − p i . Then, we calculate a feature at the position p using modified DCN, i.e., where p j ∈ {(−1, 1), (−1, 0), . . . , (1, 1)}, s k i is the cooperative weight to aggregate the K textures from the Ref image, i.e., s k i = exp( q i · k p k i )/ j∈Pi exp( q i · k j ), w j is the convolution kernel weight, ∆p j is the j-th learnable offset of ∆P l , and m j is the j-th learnable mask of M l , which can be calculated as follows, where ω is a warping function, [; ] is a concatenation operation, Conv is convolutional layers. Sigmoid and Tanh are activation functions, r is the max magnitude which is set as 10 in default, and F l is the feature of upsampled LR images at the l-th scale.

Residual Feature Aggregation
To aggregate the multi-scale LR features at different layers and the transferred texture features, we propose a residual feature aggregation module (RFA) to perform feature fusion and extraction. As shown in Fig. 4, RFA consists of CNNs and Swin Transformer layers (STL) [25] which gain much attention in many tasks [19,2,26]. Specifically, we first use a convolution layer to fuse the LR feature F l and attention features A l , i.e., F ′ l+1 = Conv(F l , A l ), where Conv is convolutional layers. Then, we use Swin Transformer and a residual connection to extract deeper features of the LR and transferred features, where the details of STL are put in the supplementary materials. At the end of RFA, we use another convolutional layer to extract the features of STL, F l+1 = Conv(F ′ l+1 ). Based on the aggregated features F L at the last scale, we synthesize SR images with a skip connection as

Loss Function
In the training, we aim to i) preserve the spatial structure and semantic information of LR images; ii) discover more texture information of Ref images; iii) synthesize realistic SR images with high quality. To this end, we use a reconstruction loss, a perceptual loss and an adversarial loss, which is the same as [48,12]. The overall loss with the hype-parameters λ 1 and λ 2 is written as: Reconstruction loss. In order to make the SR image X SR to be close to the HR ground-truth image X HR , we adopt the following reconstruction loss where ∥ · ∥ 1 is the ℓ 1 -norm. Perceptual loss. To enhance the visual quality of SR images, the perceptual loss is widely used in SR models [56,12]. The perceptual loss is defined as: where ∥·∥ F is the Frobenius norm, and V and C are the volume and channel number of the feature maps, respectively. The function ϕ i is the i-th intermediate layer in VGG19 [35], and we use the relu5 1 layer of VGG19 in the experiment. Adversarial loss. To improve the visual quality of SR images, many SR methods [17,44] introduce GANs [7,1] which have achieved good performance for SR. Specifically, we use WGAN [1] loss as follows, where D(·) is a discriminator, P SR is the distribution of the generated SR images, and P HR is the distribution of the real data.

Experiments
Datasets. In the experiment, we consider the RefSR dataset, i.e., CUFED5 [56], which consists of a training set and a testing set. Implementation details. The input LR images are generated by bicubicly downsampling the HR images with scale factor 4. For the encoders and discriminator, we adopt the same architectures as [12]. We use a pre-trained relu1 1, relu2 1 and relu3 1 of VGG19 to extract multi-scale features. we augment the training data with randomly horizontal and vertical flipping or different random rotations of 90 • , 180 • and 270 • . Besides, we also augment the training data by randomly changing different brightness, contrast and hue of an image by using ColorJitter in pytorch. In the training, we set the batch size as 9, i.e., each batch has 9 LR, HR and Ref patches. The size of LR images is 40×40, and the size of HR and Ref images is 160×160. Following the training of [12], we set the hypeparameters λ 1 and λ 2 as 1×10 −4 and 1×10 −6 , respectively. We set the learning rate of the SR model and discriminator as 1×10 −4 . For the Adam optimizer, we set β 1 =0.9 and β 2 =0.999. We provide more detailed network architectures and training details in the supplementary material.

Comparison with State-of-the-art Methods
We compare with the SISR methods (SRCNN [6], EDSR [22], RCAN [55], SwinIR [19], SRGAN [17], ENet [33], ESRGAN [44], and RankSR-GAN [53]) and RefSR methods (CrossNet [57], SRNTT [56], SSEN [34], TTSR [48], E2ENT2 [45], and MASA [27]). For fair comparisons, the above models are trained on CUFED5 training set, and tested on CUFED5 testing set, Urban100, Manga109, Sun80 and WR-SR. In this experiment, we train our model on two cases only with reconstruction loss (denoted as '-rec'), and with all loss functions.  Quantitative comparison. We provide quantitative comparisons of SR models trained with only reconstruction loss and all losses in Tables 1 and 2, respectively. In Table 1, our model has the best PSNR and SSIM on all testing sets and significantly outperforms all SISR and RefSR models. It implies that our Transformer achieves the state-of-the-arts and good generalization performance. For the SISR setting, our method performs better than the state-of-the-art SISR method [19]. It is difficult for these SISR methods to synthesize since the highfrequency information is degraded. In contrast, our model is able to adaptively discover the useful information from a reference image on the Urban100 and Manga109 datasets even if it is a random image. For the RefSR setting, our proposed DATSR significantly outperforms all methods with the help of the cooperative transfer with deformable convolution module. In Table 2, our DATSR also achieves the much higher PSNR/SSIM values than other RefSR methods with a large margin. Our DATSR trained with adversarial loss reduces PSNR and SSIM but increases the visual quality. Still, it has the best performance over all compared methods. The above quantitative comparison results on different SR paradigms demonstrate the superiority of our Transformer over state-of-the-art SISR and RefSR methods.  Qualitative comparison. The visual results of our method are shown in Figs. 5 and 6. In these figures, our model also achieves the best performance on visual quality when trained with the reconstruction loss and all loss. These results demonstrate that our proposed method is able to transfer more accurate textures from the Ref images to generate SR images with higher quality. When trained with the reconstruction loss, our medel can synthesize SR images with sharp structure. Moreover, our method is able to search and transfer meaningful texture in a local regions even if the Ref image is not globally relevant to the input image. When trained with the adversarial loss, our model is able to restore the realistic details in the output images which are very close to the HR ground-truth images with the help of the given Ref images. In contrast, it is hard for ESRGAN and RankSRGAN to generate realistic images without the Ref images since the degradation is severely destroyed and high frequency details of images are lost. For RefSR methods, our model is able to synthesize more realistic texture from the Ref images than SRNTT [56], TTSR [48], MASA [27], and C 2 -Matching [12]. For example, in the top of Fig. 6, our model is able to recover the "window" with sharper edge and higher quality than C 2 -Matching, but other methods fail to restore it even if they have a

Further Analyses
Robustness to image transformations. We analyze the robustness of our model to different kinds of image transformations. Specifically, we use ColorJitter to augment the CUFED5 testing set by randomly change the brightness, contrast and hue of Ref images into three group: small, medium and large. The detailed settings are put in the supplementary materials. In Fig. 7, our model is more robust than C 2 -Matching [12] under different image transformations. Note that the medium and large transformations are not included during training but our model still has superior performance.
Effect on type and number of Ref images. We test our model on different Ref images, such as extreme images (i.e., may have only one color or noise without any information) and random images from different testing sets. In Fig.  8, our method has robust performance and high visual quality even if the Ref images have no useful texture information. In addition, our model has better performance when increasing #Ref images in Fig. 9. Table 3 shows the results of four similarity levels ("L1" to "L4") where L1 is the most relevant level. Our method achieves the best performance across all similarity levels.
Comparisons with multi-RefSR methods. We compare our model with multi-RefSR methods, i.e., CIMR-SR [47] and AMRSR [32]. Note that these multi-RefSR methods are trained with a collection of reference images. In Table  4, our model trained with single reference image performs better than CIMR-SR and AMRSR with many reference images, which further demonstrate the superiority of our proposed DATSR.

More Evaluation Results
Perceptual metric. We further use the perceptual metric LPIPS [52] to evaluate the visual quality of the generated SR images on the CUFED5 and WR-SR testing sets. Recently, this metric is also widely used in many methods [28,29]. In general, smaller LPIPS corresponds to the better performance for RefSR. As shown in Table 5, our model achieves smaller LPIPS than C 2 -Matching. Thus, our model generates SR images with better quality than C 2 -Matching.
User study. To further evaluate the visual quality of the SR images, we conduct the user study to compare our proposed method with previous state-of-the-art methods, including SRNTT [56], TTSR [48], MASA [27] and C 2 -Matching [12] on the WR-SR testing set. The user study contains 20 users, and each user is given multiple pairs of SR images where one is our result. Then, each user chooses one image with better visual quality. The final percentage is the average user preference of all images. In Fig. 10, over 80% of the users prefer that our results have better quality than existing RefSR methods.

Discussion on Model Size
To further demonstrate the effectiveness of our model, we also show the comparison of model size (i.e., the number of trainable parameters) with the state-of-theart model (i.e., C 2 -Matching [12]) in Table 6. Our model has a total number of 18.0M parameters and achieves PSNR and SSIM of 28.72 and 0.856, respectively. The results demonstrate that our proposed model outperforms C 2 -Matching with a large margin, although our model size is higher than this method. The part of our model size comes from the Swin Transformer in the RFA module. More discussions of other RefSR models are put in the supplementary materials.

Ablation Study
We first investigate the effectiveness of RDA and RFA in Table 7. Specifically, we replace the texture transfer method in RDA with a feature warping based on the most relevant correspondence, and replace RFA with several convolutional neural networks (CNNs). The model with feature warping or CNNs is worse than original model with RDA or RFA. Therefore, RDA is able to discover more relevant features especially when the correspondence is not inaccurate.
For RFA, our model has better performance than the directly using simple CNNs. Nevertheless, with the help of RDA, training with CNNs still outperforms C 2 -Matching with large margin. Therefore, it verifies that the effectiveness of RFA and it is able to aggregate the features at different scales. More discussions on ablation studies are put in the supplementary materials.

Conclusion
In this work, we propose a novel reference-based image super-resolution with deformable attention Transformer, called DATSR. Specifically, we use texture feature encoders module to extract multi-scale features and alleviate the resolution and transformation gap between LR and Ref images. Then, we propose reference-based deformable attention module to discover relevant textures, adaptively transfer the textures, and relieve the correspondence mismatching issue. Last, we propose a residual feature aggregation module to fuse features and generate SR images. Extensive experiments verify that DATSR achieves the stateof-the-arts performance as it is robust to different brightness, contrast, and color between LR and Ref images, and still shows good robustness even in some extreme cases, when the Ref images have no useful texture information. Moreover, DATSR trained with a single Ref image has better performance than existing Multi-RefSR methods trained with multiple Ref images.
Acknowledgements. This work was partly supported by Huawei Fund and the ETH Zürich Fund (OK).


BRATECA (Brazilian Tertiary Care Dataset): a Clinical Information Dataset for the Portuguese Language

Computational medicine research requires clinical data for training and testing purposes, so the development of datasets composed of real hospital data is of utmost importance in this field. Most such data collections are in the English language, were collected in anglophone countries, and do not reflect other clinical realities, which increases the importance of national datasets for projects that hope to positively impact public health. This paper presents a new Brazilian Clinical Dataset containing over 70,000 admissions from 10 hospitals in two Brazilian states, composed of a sum total of over 2.5 million free-text clinical notes alongside data pertaining to patient information, prescription information, and exam results. This data was collected, organized, deidentified, and is being distributed via credentialed access for the use of the research community. In the course of presenting the new dataset, this paper will explore the new dataset’s structure, population, and potential benefits of using this dataset in clinical AI tasks.

Introduction
Decision making in healthcare scenarios has been a topic of increasing interest in the field of artificial intelligence (Shamout et al., 2021;Si et al., 2021). Studies into methods for predicting diseases (Song et al., 2018;Xu et al., 2018), mortality (Xu et al., 2018), lengthof-stay (Song et al., 2018;Xu et al., 2018), admission (Liu et al., 2019), and interventions (Suresh et al., 2017) have become more common with the widespread use of EHRs (Electronic Health Records) in hospitals worldwide, which in turn led to efforts to deidentify this information, as it is sensitive and extremely personal, in order to make it available for use in related research (Shamout et al., 2021). These efforts have resulted in several clinical datasets, most of which were constructed from sources in the English language. The MIMIC (Medical Information Mart for Intensive Care) collection, composed of data extracted from the Beth Israel Deaconess Medical Center in the United States, is a part of PhysioNet (Goldberger et al., 2000) and is the foremost example of such datasets in computational medicine and focuses on intensive care patients. Many papers (Song et al., 2018;Xu et al., 2018;Liu et al., 2019;Suresh et al., 2017) make use of this collection to build training and testing datasets for deep learning architectures. MIMIC-IV (Johnson et al., 2020), the latest edition of the collection, possesses comprehensive information about each patient, including laboratory measurements, administered medication, documented vital signs, etc. The present work's goal is to introduce a large collection of clinical data akin to the MIMIC collection but for Brazilian Portuguese instead. Being focused on clinical notes in the Portuguese language, it can be used for any project focusing on Brazilian clinical scenarios. This new dataset, henceforth referred to as BRATECA (BRAzilian TErtiary CAre dataset), boasts more than 400 million words across over 2.5 million free-text clinical notes from over 70,000 individual admissions in 10 different hospitals located in two Brazilian states. The dataset also possesses patient, prescription, and exam information for these admissions, when available. This data is collected, deidentified and managed by the Institute for Artificial Intelligence in Healthcare 1 , a non-profit startup from Brazil composed of an interdisciplinary team of data scientists and practicing healthcare professionals such as pharmacists and physicians that develop smart systems for clinical pharmacy. The dataset has been made available by them for credentialed access.

Related Work
One of the most widely used clinical datasets is MIMIC. It has several versions, and its most current iteration, MIMIC-IV, is separated into 6 modules: core, hosp, icu, ed, cxr, and note. The core module is composed of patient demographics, hospitalization records, and ward stay records. The hosp module is composed of data recorded during the patient's hospital stay such as lab measurements, medication administration and prescription, billing information, etc. The icu module is composed of data taken from patients in intensive care units (ICUs), and include intravenous and fluid inputs, patient outputs, procedures, date and time information, etc. The ed module is composed of data from emergency department (ED) patients, and includes reason for admission, triage assessment, vital signs, etc. The cxr module contains chest x-ray (CXR) images from ED patients from multiple viewpoints. Finally, the note module contains patient's deidentified free-text clinical notes for hospitalization, although this module is not yet available to the public. Another example is the United Kingdom's National Health Service's (NHS) comprehensive dataset collection 2 . The data is collected in order to support the analysis of specific policies of interest as well as the effects of particular policy initiatives, and it is separated into several different datasets, each with a different focus and different kinds of data. A more task-focused example of English language clinical dataset can be found in the National NLP Clinical Challenges (n2c2) datasets. These challenges have been proposed since 2006, starting with the i2b2 project, n2c2's predecessor. These two series of challenges have presented datasets for a variety of tasks, such as deidentification, obesity prediction, coreference, temporal relations, heart disease, clinical semantic textual similarity, and family history extraction. The current edition, n2c2 2022 3 , proposes three tracks: Contextualized Medication Event Extraction; Extracting Social Determinants of Health; and Progress Note Understanding: Assessment and Plan Reasoning. Task-specific datasets were released alongside each of these challenges, though some, such as the current challenge's third track, make use of already available resources (MIMIC-III in this case) when they are appropriate for the proposed task. However, these are English language datasets extracted from hospitals in certain anglophone countries, and do not conform to the clinical realities of Brazil. It is thus important to gather national data for local research projects which may be able to positively impact Brazilian public health. The development of national clinical resources has started in earnest in recent years, with work such as SemClinBR (e Oliveira et al., 2020), a dataset with 1000 clinical notes annotated with over 65,000 entities and over 11,000 relations. The dataset was manually annotated and may be used for a variety of tasks, such as clinical named entity recognition and negation detection. It bears more resemblance to the n2c2 challenge datasets than to MIMIC. BioBERTpt (Schneider et al., 2020) is a fine-tuned BERT model trained on clinical EHR texts as well as texts from the biomedical literature. It has three versions, each trained with a different corpus. The uk/data-and-information/ data-collections-and-data-sets/data-sets 3 https://n2c2.dbmi.hms.harvard.edu/ 2022-challenge first was trained with more than 2 million clinical notes from Brazilian hospitals collected between 2002 and 2018. The second with titles and abstracts from Portuguese biomedical scientific papers published in PubMed and Scielo. A third version combining both corpora into one was also trained. The clinical note corpus does not seem to have been made available after its use in training the models. The literature also covers a Brazilian healthcare image dataset, the labeled chest X-ray dataset BRAX (Reis et al., 2022). Although it is not a language resource, that dataset is nonetheless an example of a Brazilian healthcare dataset, and it is similar to MIMIC's CXR (chest X-ray), except that the images are not complemented by text-based healthcare resources like MIMIC's. Another example of a Portuguese-language healthrelated dataset was developed by de Melo and Figueiredo (2020). Their Twitter-based dataset is composed of nearly 4 million tweets and about 18,000 news articles related to COVID-19 in Brazil. It has a different domain from the other datasets presented thus far and so has a different overall purpose, being more focused on public discourse and sentiment about public health issues rather than clinical information.

BRATECA
BRATECA contains 73,040 admission records of 52,973 unique adults (18 years of age or older) extracted from 10 hospitals located in two Brazilian states. Amongst those admissions, several are associated with specialty treatment wards, as follows: publicly funded wards (12,096 admissions total); intensive care wards (4,666 admissions total); obstetrics wards (5,550 admissions total); COVID-19 wards (1,714 admissions total); surgical wards (25,004 admissions total); emergency wards (37,392 admissions total); and ambulatory wards (3,107 admissions total). The remaining 8,674 admissions associated with any specialty wards. The median patient age is 54 (Q1 = 38, Q3 = 68), 41.3% of the patients are male, 70.7% are identified as white, 3.8% are identified as mixed, 3.8% are identified as black and 0.2% are identified as yellow, and the mortality rate of patients is 6.5%. Each admission is paired with laboratory exam results (2,374,807 total), prescriptions and their itemized contents (519,318 total), and clinical notes (2,849,572 total). An interactive dashboard has been created to present some details of BRATECA and is linked in the project's GitHub page 4 . Table 1 presents statistics for each admission type.

Classes of Data
BRATECA is composed of descriptive data, laboratory data, medication data, intervention data, and clinical notes. Descriptive data includes patient specific information such as dates of birth, admission and discharge, skin color, height, weight, and reasons for dis-  Table 1: Rows present each the following information, from top to bottom: Median age (Q1 through Q3) per admission type; Median number of laboratory results per patient per admission type; Median number of prescriptions per patient per admission type; Median number of clinical notes per patient per admission type; Percentage of male patients per admission type; mortality percentage per admission type; percentages for patient skin color identification (W is white, B is black, M is mixed, Y is yellow and NI is no information). Columns each present one type of ward. Wards deemed "normal" are those that do not fall into any of the other categories. Note that a single admission may have a patient move wards one or more times, and a single ward may belong to more than one category.
charge. Laboratory data include data on various laboratory exam results for patients. Medication data includes prescription items, as well as dosage, frequency, and other such administration details specific to each patient and prescription. Intervention data includes notes on whether there were pharmacist interventions on specific prescriptions that may have been mistakenly administered, as identified by the Institute for Artificial Intelligence in Healthcare's NoHarm.ai clinical pharmacy AI system (D. P. dos Santos et al., 2021). Notes are free-text clinical notes describing a patient's evolving hospital admission.

Development Methods
BRATECA is an edited and reorganized version of the Institute for Artificial Intelligence in Healthcare's own internal Brazilian tertiary care information database and is intended to be a public 5 edition for use in machine learning research. For this purpose, certain data tables deemed most useful at the time of extraction were reorganized into the 5 datasets of BRATECA. This section presents the process of extraction and deidentification of the database's information into the format presented in Section 5.

Dataset Organization
The Institute for Artificial Intelligence in Healthcare's database is centered around its prescription tables. This resulted in only admissions with prescriptions being extracted, as the prescription tables contained ward information and were the best way to ascertain that only 5 Note that BRATECA is property of the Institute for AI in Healthcare and only credentialed access is allowed, but it is freely available for research use. adult patients from the desired wards were extracted from the database. Beyond those requirements, only admissions which both began and ended during a delimited time period of nine months were extracted. This time period was set to sometime between 2020 and 2021, but this will not be specified so as to further enhance patient privacy. All admissions that fit within the presented parameters had their IDs extracted and used to gather related data from the database and create the 5 separate but interconnected datasets: Admission, Exam, Clinical Note, Prescription, and Prescription Item. These datasets are further described in Section 5. The SQL scripts used to extract the data are available in the project's GitHub page 6 .

Deidentification
Though most columns in the datasets provide the exact information present in the original database, some had to be modified to further protect patient's sensitive information and attempt to prevent reverse engineering of identities from the provided data. All names in BRATECA's free text notes were deidentified using state-of-the-art deep learning methods (Bi-LSTM-CRF) (Akbik et al., 2018). Two corpora and three language models were evaluated on a Named Entity Recognition (NER) task focused on person names to evaluate which combination delivered the best performance. The experiments revealed that using domain-specific corpora (focused on deidentification of clinical notes) and a contextualized embedding stacked with word embeddings achieved the best results: an F-measure of 0.94 and Recall of 0.95 (Santos et al., Figure 1: A simple example timeline of an admission, including recorded time of admission, laboratory examination, prescription administration, clinical note writing, and discharge. The two labels represent two instances where events were logged at the same time. In these cases, 15 and 8 exam results were logged simultaneously at two separate points in the timeline. 2021). Dates present in the free text notes were also removed, though not using NER but rather regular expressions. The date removal script is available on the project's GitHub page 7 . Furthermore, all dates not part of free-text notes were shifted randomly 5 to 10 years forward. Dates referring to the same admission were shifted the same amount of days forward (i.e. if admission "1" of the Admission dataset was shifted 100 days forward, all dates of all entries in the other 4 datasets which refer to admission "1" in their Admission ID field were shifted 100 days forward as well). This was done in order to maintain timeline coherence within the same admission. Note that multiple admissions of the same patient may not be in chronological order and do not maintain any sort of temporal relation in order to more thoroughly deidentify such patients. All internal database IDs, such as those for Patient ID or Admission ID, were also deidentified. Each was assigned a random numerical ID, congruent between datasets (i.e. if Admission ID "123456" is assigned the new ID "789" in the Admission dataset, the Admission ID "123456" was also assigned "789" whenever it appeared in the other 4 datasets). Finally, ward information 8 was generated using the actual names of the wards of the hospitals from which the information was collected. Ward names were replaced with the aforementioned labels in order to better prevent hospital identification while maintaining some of the more relevant information. The generation was 7 See footnote 4 8 Public, IC, Obstetrics, COVID-19, Surgical, Emergency and Ambulatory. See the Prescription row of Table 2 for further details. performed with the help of an active healthcare professional.

Code Availability
All of the code associated with BRATECA (algorithms, sample datasets, language models, and experiments) is available on the project's GitHub page 9 for replicability purposes.

Data Records
BRATECA is composed of 5 datasets in the CSV (Comma Separated Values) format. These are as follows: Admission, a dataset of every individual admission, which includes patient demographic data; Exam, a dataset of exams and their respective results performed for each admission; Prescription, a dataset of prescription headers, which includes information such as patient/admission ID for the patient/admission which received the prescription, pharmacy assessments, prescription date, expiration date, ward information, whether the prescription includes special medication such as controlled substances, intravenously administered drugs (IV drugs), and antibiotics; Prescription Item, a dataset of prescribed medications which includes details of each prescribed medication, including name, dosage, and information on how the medication is to be administered, with each entry of this dataset being directly related to a prescription header in the Prescription dataset; and Clinical Note, a dataset of free-text clinical notes on details of the patient's stay and treatment. A simple example of a patient timeline which shows details from all datasets in conjunction can be seen in Figure 1.

Dataset
Column Description Column Description

Hospital ID
The identification code for the hospital from which the data originated.

Patient ID
The identification code for the patient for whom the admission was registered.

Admission ID
The identification code for the admission to which the information belongs.

Hospital ID
The identification code for the hospital from which the data originated.

Patient ID
The identification code for the patient for whom the admission was registered.

Admission ID
The identification code for the admission to which the information belongs. Exam Name Name of the exam that was performed.

Exam Date
Date the exam was performed Value Numerical value of the result of the exam.

Unit
Unit of measurement the exam's Value is in.
Clinical Note

Hospital ID
The identification code for the hospital from which the data originated.

Patient ID
The identification code for the patient for whom the admission was registered.

Admission ID
The identification code for the admission to which the information belongs.

Note Date
Date the note was written.

Note Text
The contents of the note.

Hospital ID
The identification code for the hospital from which the data originated.

Patient ID
The identification code for the patient for whom the admission was registered.

Admission ID
The identification code for the admission to which the information belongs.

Prescription ID
The identification code for the prescription to which prescription items are associated.

Prescription Date
Date the prescription note was written.

Pharmacy Assessment
Whether the prescription was revised by a pharmacist.

Expiration Date
Prescription expiration date.

Assessment Date
Date the pharmacy assessment was performed.

Allergy
Whether patient is allergic to one or more of the prescribed medications.

Prescription Score
Score generated by artificial intelligence (the higher the score, the more unusual the prescription).

Alerts
Prescription alerts. A complete list of alerts is shared in the documentation.

Score One
The quantity of prescription items given a "1" score by the AI.

Antibiotics
Number of antibiotics prescribed.

Score Two
The quantity of prescription items given a "2" score by the AI.

High Alert
Number of high alert medication prescribed.

Score Three
The quantity of prescription items given a "3" score by the AI.

Controlled
Number of controlled medication prescribed. Tube Number of IV drugs prescribed.

Not Default
Number of non-standard medications prescribed.

Different Drugs
Number of prescribed medications not previously reviewed by a pharmacist.

Alert Exams
Alerts related to exams. Examples can be found in the documentation.

Interventions
Number of interventions related to the prescription.

Complication
Number of complications detected in clinical notes related to the prescription. Public Whether or not the prescription is for a publicly funded ward.

IC
Whether or not the prescription is for an Intensive Care ward.

Obstetrics
Whether or not the prescription is for a obstetrics ward.

COVID-19
Whether or not the prescription is for COVID-19 ward. Surgical Whether or not the prescription is for a surgical recovery ward.

Emergency
Whether or not the prescription is for an emergency ward. Ambulatory Whether or not the prescription is for an ambulatory ward.

Hospital ID
The identification code for the hospital from which the data originated.

Patient ID
The identification code for the patient for whom the admission was registered.

Admission ID
The identification code for the admission to which the information belongs.

Prescription ID
The identification code for the prescription to which prescription items are associated.

Drug Name
Name of the drug. Dosage Dosage of each administration.

Daily Frequency
Number of times a drug is administered per day.

Administration Route
Route of drug administration.

Note
Medical observations related to the prescription.

Normalized Dosage
Dose converted to a single numerical unit.

Time
Time each dose is to be administered.

Source
Whether it is nutrition, a drug, a procedure drug or a solution.

Suspension Date
Date the medication is to be suspended.

(Solution) Group
Group to which the solution belongs.

(Solution) at Medical Discretion
Medical observations related to the prescribed solution.
(Solution) Steps Frequency of solution adminitration.

(Solution) Hour
Time each solution dose is to be administered.

(Solution) App Time
How long a solution is to be administered for.

(Solution) Dosage
The dosage of the solution.

(Solution) Unit
The unit of measurement of the dosage.

Administration Period
The period during which the item is to be administered.

Allergy
Whether the patient is allergic to the prescription item.

Tube
Whether the prescription item is administered intravenously.

(Intervention) Error
Intervention considered a prescription error.

(Intervention) Cost
Intervention that generated a reduction of costs. All datasets have IDs that are used for identification of relations between entries in each file. These are: Hospital ID, the identification for the hospital from which the raw data was collected; Patient ID, the ID for a given patient in the database; Admission ID, the ID for the patient's admissions, of which a single patient might have many; and Prescription ID, specific to the Prescription and Prescription Item datasets, which identifies prescription items as belonging to specific prescriptions.

Datasets
The datasets were developed in the way described above so that they can be used separately as well as in conjunction. Each is composed of several columns from tables in the original database, organized for ease of use. The information in each of the 5 datasets are presented in Table 2.
6. Usage Notes 6.1. Data Access As mentioned previously, BRATECA is distributed by the Institute for Artificial Intelligence in Healthcare through Physionet credentialed access. In order to receive access, the researcher must complete the following steps: 1. sign in to and confirm your identity in the Physionet platform 2. complete a course on protecting human research participants; 3. if the requester is a student, their supervisor must also agree to the terms of confidentiality; 4. access the BRATECA page in Physionet 10 and request access to the dataset; 5. wait for approval by the Institute for Artificial Intelligence in Healthcare.
Once the process is complete, and if the request is accepted, the researcher will be granted access to the dataset files.

Example Usage
There are many tasks which could benefit or even require datasets such as BRATECA. Prediction tasks, such as those mentioned in Section 1, can use these datasets for training purposes. Mortality prediction can use discharge information as mortality annotation, for example. Researchers with access to the original database have already published several papers with the information which is to be released in BRATECA. Santos et al. (2021), for example, used state-of-the-art methods to identify and remove names from clinical texts. These were the methods were used to deidentify all free-text notes made available as part of BRATECA, as mentioned in Section 4. • A machine learning-based clinical decision support system to identify possible drug intervention; • Detection of Drug-Induced Liver Injury; • Trends in the use of corticosteroids during the Pandemic.
Finally, several other usages of the dataset are being investigated or set to be explored in the near future. The large amount of free text notes, for example, permits the training of domain-specific language models with word embedding architectures such as Word2Vec (Mikolov et al., 2013) and fastText (Grave et al., 2017), and also contextual embedding models such as ELMO (Peters et al., 2018) and BERT (Devlin et al., 2019). Embeddings like these can be even more specific, using only certain parts of the data, such as limiting training to texts about elderly patients or intensive care patients. Another avenue of research being explored is the use of the information to create real-time digital twins of patients by utilizing representation learning technology. These digital twins could be used to predict patient developments and aid medical workers keep track of the most important information for each of their patients via alerts, data organization, and information retrieval (Shamout et al., 2021).

Continuous Development
BRATECA will undergo continuous maintenance and development to ensure the high quality of the data made available for researchers to further the national public good. The maintenance team will encourage the user community to aid in this important endeavor as well, so as to continue to improve the quality of available clinical data for Portuguese language research. These continuous efforts will, for example, be aimed at ensuring that the data is fully deidentified. Researchers granted access to the data will be asked to report any deidentification errors they might find so that the patient data is kept as private as it can be while still being useful to data scientists and machine learning experts. The team will also look into creating annotated subsets from the free-text notes for several tasks relevant for clinical text research, such as named entity recognition, text labeling, coreference, semantic textual similarity, and others. The user community will also be encouraged to allow any such annotated subsets they might themselves develop to be distributed via credentialed access by the Institute for Artificial Intelligence in Healthcare alongside the original version of BRATECA. Finally, the team will endeavor to develop further versions of BRATECA by adding more data as it becomes available, and making improvements to the dataset according to community feedback. These future versions will be released alongside the original version of BRATECA, similarly open to credentialed access.

Ethical Concerns
BRATECA has been deidentified according to the Health Insurance Portability and Accountability Act (HIPAA) standards using structured data cleansing and date shifting. The NoHarm.ai system, developed by the Institute for Artificial Intelligence in Healthcare, gathers no identifiable information from patients. The data used in the experiments we conducted for this article came from a research project developed with several hospitals in Brazil. Also, all data sharing was approved by each hospital participating in that research. Ethical approval to use the hospitals' datasets in this research was granted by the National Research Ethics Committee under the number 46652521.9.0000.5530.


CroMo: Cross-Modal Learning for Monocular Depth Estimation

Learning-based depth estimation has witnessed recent progress in multiple directions; from self-supervision using monocular video to supervised methods offering highest accuracy. Complementary to supervision, further boosts to performance and robustness are gained by combining information from multiple signals. In this paper we systematically investigate key trade-offs associated with sensor and modality design choices as well as related model training strategies. Our study leads us to a new method, capable of connecting modality-specific advantages from polarisation, Time-of-Flight and structured-light inputs. We propose a novel pipeline capable of estimating depth from monocular polarisation for which we evaluate various training signals. The inversion of differentiable analytic models thereby connects scene geometry with polarisation and ToF signals and enables self-supervised and cross-modal learning. In the absence of existing multimodal datasets, we examine our approach with a custom-made multi-modal camera rig and collect CroMo; the first dataset to consist of synchronized stereo polarisation, indirect ToF and structured-light depth, captured at video rates. Extensive experiments on challenging video scenes confirm both qualitative and quantitative pipeline advantages where we are able to outperform competitive monocular depth estimation method.

Introduction
Modern vision sensors are able to leverage a variety of light properties for optical sensing. Common RGB sensors, for instance, use colour filter arrays (CFA) over a pixel sensor grid to separate incoming radiation into specified wavebands. This allows a photosensor to detect wavelengthseparated light intensity and enables the acquisition of familiar visible spectrum images. Wavelength is however only one property of light capable of providing information.
Light polarisation defines another property and describes the oscillation direction of an electromagnetic wave. While the majority of natural light sources (e.g. the sun) emit unpolarised light, consisting of a mixture of oriented oscillations, surface reflection from non-metallic objects can  input. Uni-modal model training of p2d [11] and the monodepth2 architecture [22], compared with cross-modal training (Ours).
linearly polarise the light. Such polarised light then contains surface structure information, retrievable using analytic physical models [8]. This information can be used to harness the depth cues offered by this light property. Polarimetric imagery is a passive example for depth estimation. Passive sensors have acceptable resolution and dense depth however there exist well understood capture situations that prove challenging (e.g. textureless surface regions).
However further known properties of light (i.e. speed) provide yet more information. Indirect Time-of-Flight (i-ToF) cameras are active light sensors and use a pulsed, near infrared light source to measure object and surface distance. Further active sensors use structured-light and these emit known infrared patterns and use stereoscopic imaging to measure the distance to the surface. While i-ToF and structured-light cameras have clear advantages, such as the ability to function in low-light scenarios and good short range precision, they are susceptible to specular reflections, ambient light and range remains limited.
We argue that novel combination of active and passive light sensors offers new possibilities. We can exploit such a combination to take advantage of the discussed, modalityspecific strengths and weaknesses. We observe that (1) differing visual modalities offer information cues about com-plimentary aspects of the world and (2) there exist clear trade-offs between the complexity of capture sensor setups and the resulting data diversity and quality, accessible for supervision signals. This motivates us to systematically investigate these considerations and provide insight into training data capture design decisions and the related pay-offs. Our study results in the proposal of a framework capable of exploiting available supervision signals and is tailored to benefit from the particular strengths of unique modalities.
We instantiate our ideas by bringing together the physical understanding of Polarisation and i-ToF in a data driven fashion. In practice this affords an inference pipeline that estimates depth from a single polarisation image. We train a convolutional neural network (CNN), with cross-modal fusion using differentiable physical models. We establish a dataset comprising Ground Truth depth obtained via Multi-View Stereo (MVS) reconstruction [52] that enjoys access to information rich, full video sequences. We carry out extensive experimental work to establish the efficacy of our proposed monocular depth estimation strategies. Our contributions can be summarised as: 1. Novel multi-modal method. We propose a multi-modal training approach that allows for monocular depth estimation from polarisation images. We propose (i) differentiable analytic formulae that define modal-specific loss terms, (ii) cross-modal consistency joint-training towards improved real-world depth estimation from a single polarisation image, (iii) architectural components that increase predicted depth sharpness (see Fig. 1). 2. CroMo dataset and training modalities study. We provide a systematic analysis of the benefits afforded when multiple image modalities are available at training time, for monocular depth estimation. Investigation and exposure of improvements are enabled by the unique Cross-Modality video dataset 1 . Our multiple hardwaresynchronized cameras capture, for the first time, stereo polarisation (Pol), indirect Time-of-Flight (i-ToF) and structured-light images from active sensing. The remaining sections of the paper are thus organised: Sec. 2 provides brief review of depth estimation with respect to relevant modalities and previous work considering multiple information signals. Sec. 3 presents our model capable of monocular depth estimation from polarisation imagery and our cross-modal training procedure. In Sec. 4 we introduce CroMo, our novel multi-modal dataset, Sec. 5 reports experimental work validating our contributions and Sec. 6 provides discussion and future research issues.

Related Work
To the best of our knowledge this is the first work to study end-to-end monocular depth inference, utilising cross-modal information from Time-of-Flight (i-ToF), ac-1 Dataset is available at: https://cromo-data.github.io/ tive stereo and polarisation modalities during training. We briefly review the literature most closely related to the main components of our investigation and proposed framework.

Monocular depth estimation
Estimating depth from a single image constitutes a hard, ill-posed problem. Pioneering work on supervised monocular depth estimation [42] used synthetic samples during training. Synthetic data was also previously used in conjunction with stereo network distillation [25] for this task. To improve accuracy and convergence speed, [17] introduce a spatially-increasing discretisation. However, acquiring ground truth depth data remains a difficult task [20].
To overcome the difficulty of collecting accurate ground truth signal, multiple works [19,61] investigate a consistency loss by leveraging stereo imagery during training, towards self-supervision. While being undoubtedly path-breaking, the initial methods suffered from a nondifferentiable sampling step. Godard et al. [21] formulated a fully-differentiable pipeline with left-right consistency checks during training and have also explored the temporal components [22], even in challenging setups such as night scenes [55]. These methods predict depth with RGB input, while we utilise polarisation images. Monocular Polarisation Previous work use monocular polarisation imagery to recover depth. One route to overcome Shape from Polarisation (SfP) ambiguities is to use orthographic camera models to express polarisation intensity in terms of depth [64]. Atkinson et al. [8] compute depth without knowing the light direction through a non-linear optimization framework and yet assume fully diffuse surfaces. Linear systems have also been constructed for the task [53] by adding shape from shading equations. While theoretically interesting, the orthographic assumption has restricted their application to synthetic lab environments. Learning based Polarisation Due to lack of reasonablysized datasets, only a limited number of works focus on learning with polarisation. Ba et al. [9] provide polarisation images together with a set of plausible inputs from a physical model to estimate surface normals. The work of [34] apply polarisation for instance segmentation of transparent objects and [37] learn de-glaring of images with semitransparent objects. Recently, Blanchon et al. [11] extended the work of [22] with complementary polarimetric cues. In contrast to them, we invert a physical model to enable self-supervision through consistency cycles and additionally study the benefit of co-modal i-ToF information. Learning based i-ToF i-ToF sensors acquire distance information by estimating the time required for an emitted light pulse to be reflected [65]. Sensors measure either the time (direct) or the phase (indirect) difference between emitted and received light. The modality enjoys high precision for short range distances [27], yet suffers from limited spatial resolution and noise [15], which constitute challenging fac-tors for any learning-based approach. Obtaining reliable signals from specular surfaces is difficult and inherent Multi Path Interference (MPI), often manifests as noisy measurements and artifacts. Synthetic training is also explored for raw i-ToF input data in end-to-end fashion [5,24,57]. However, the ability to account for real world domain shifts is limited. In [4] a GAN is employed towards addressing such domain adaption issues on a limited dataset. i-ToF depth improvement MPI can be considered a critical issue and error mitigation has been the focus of a body of work [60]. Two-path approximations [23] have been used within optimization schemes [14,35] and multiple frequencies are used to constrain the problem [16]. Kadambi et al. [33] propose a hardware solution to address scenes with translucent objects and a number of scholars incorporate light transport information to correct for MPI [2,26,44,46].

Depth with multiple sensors
Depth completion has been carried out via combining multiple input modalities, for example, a sparse but accurate LiDAR signal in combination with RGB [58]. It is difficult to address sparse signals with CNNs [41] and LiDAR sensors can produce problematic artifacts resulting in unreliable Ground Truth depth estimates [39]. One strategy towards removing dependence on this form of supervision are self-supervision cues however these fall behind supervised pipelines in terms of accuracy [40]. i-ToF and x Confidence-based combination of i-ToF depth and classical RGB stereo is explored with the network architecture of [3] and a semi-supervised approach for this combination is explored by [47] in a generic framework. While these approaches improve upon the individual depth estimates, they rely on a late fusion paradigm. Son et al. [54] use a robotic arm to collect 540 real training images of short range scenes with structured light ground truth.
By inserting micro linear polarizers [45] in front of a photo-receiver chip, Yoshida et al. [62] build an i-ToF sensor capable of acquiring both i-ToF depth and polarisation scene cues. Combination of both the absolute depth (i-ToF) and relative shape (polarisation cues) allowed reconstruction of depth for specular surfaces. While this pipeline requires i-ToF and polarisation input to solve an optimization problem, we alternatively explore cross-modal selfsupervised training and single image inference. Depth from multi-view Polarisation Another route to predict depth is the use of more than one polarisation image [7] which enables methods based on physical models. An RGB+Polarisation pair can provide sharp depth maps with stereo vision [66]. Other methods [12] use more than two polarisation images. Despite the sharpness of the results, the difficulty to acquire multi-view polarisation images is still a major hurdle. Atkinson et al. [6] combine polarisation methods with photometric stereo. Two images of a scene, from an identical view point yet with dif-ferent light exposures, are leveraged. An extension dealing with mixed reflectivity is established via a combined photometric-polarisation linear system in [38] and Garcia et al. [18] solve for polarisation normals using circularly polarised light. Traditional multi-view methods also benefit from polarisation. Miyazaki et al. [43] recover surfaces of black objects using polarisation physics and space carving. Depth refinement with Polarisation Consumer depth estimation tools progress significantly in recent years however their predictions are noisy and lack details. Using polarisation cues, [32] enhance sharper depth maps from RGBD cameras by differentiating their depth maps to resolve polarisation ambiguities and perform mutual optimization.
Despite clear improvements in monocular depth estimation methods, their performance remains bounded by the chosen modality hence calling for multi-modal depth estimation. Our method alleviates this problem with a learning based approach. During training we leverage complementary modalities such that our model can compensate the drawbacks of the single modality used at inference time.

Method
Our multi-modal monocular depth investigation leads to a new model architecture that accounts explicitly for prediction blur and introduces two novel analytic losses. We discuss these components in the following sections.

Architecture
Our architecture employs multiple encoder-decoder networks illustrated in Fig. 3a. We observe that monocular depth estimation methods often incur blurry image predictions and we address this problem by introducing architectural components that account for prediction blur. Firstly convolutions in our encoders are coupled with gated convolution. Our network then composes a traditional U-Net [51] with skip connections and the gated convolutions [63]. The encoder utilises a ResNet [28] style block, while the decoder is a cascade of convolutions with layer resizing.
Secondly, drawing on the fact that Displacement Fields (DF) can be utilised to aid sharpness [49], we estimate a DF using a self-supervised sharpening decoder. Depth pixels with strong local disparity have values redefined to mirror a nearest neighbour that does not exhibit strong local disparity. Groundtruth (GT) displacement fields can thus be defined for each predicted depth during training ("on-thefly"), guiding our dedicated displacement field prediction. We inspect predicted depth with and without our DF strategy and observe significant improved sharpness, most evident when employing 3D visualisations (Fig. 4).

Loss Formulation
Our study considers multiple modalities and various sensor configurations at training time. We explore several loss terms to exploit our unique setting (see Tab   in our training procedure are enabled through both coordinate frame projections (P1, P2) and analytic transforms (A1, A2) of individual network (N1, N2) outputs (see Figs. 3a, 3b). We firstly process input modalities individually using distinct networks. These ingest i-ToF correlation and left polarisation images respectively and output initial depth maps. We propose two analytic losses, derived from properties of i-ToF and polarisation, to train and link the networks. We train the i-ToF module without ground truth and also leverage the available multi-modal information through image recovery via related analytical formulae (A1, A2). Strategically similar to previous work [13], at inference time we require only a single modality (in our case polarisation), and can discard network N1 completely.
Terms L corr→pol and L corr evaluate discrepancies between each input image and respective recovered images, obtained using auxiliary and final depth maps (Fig. 3b). Our individual branches share information through the loss term L corr→pol . Explicitly, we recover a polarisation image from an auxiliary depth map and then project this, using projection P1, to the polarisation sensor frame of reference via the final depth map D pol . Finally our third loss term L stereo is used to train the polarisation network (N2) by comparing the right polarisation image, projected using the predicted depth D pol , with the left polarisation image. We next provide details of our analytical formulae for image recovery and the loss terms that enable our training procedure.
Depth to polarisation (A2) Polarisation cameras capture polarised intensity along directions ϕ pol . The measured intensity is given by [66] where ϕ pol is the polariser angle, i un is the intensity of unpolarised light, ρ is the degree of linear polarisation and φ is the Angle of Polarisation (AoP). The polarisation parameters ρ ∈ {ρ s , ρ d } and φ ∈ {φ s , φ d } depend on local reflection type, either diffuse (d) or specular (s) as follows: with θ ∈ [0, π/2] the viewing angle and η the object refractive index, typically 1.5, and if the pixel is diffuse if the pixel is specular The π-ambiguity is denoted as [π] in (3), and α denotes the azimuth angle of the surface normal n. Azimuth angle α and viewing angle θ are obtained as with v the viewing vector defined as the vector pointing toward the camera center from the 3D point P (x, y) corresponding to pixel (x, y) with depth d(x, y) and n the outward pointing normal vector, defined as the cross product of the partial derivatives with respect to x and y [66]: with f x , f y , c x , c y the camera intrinsics.
Hence, from a given depth map d, one can compute the azimuth angle α and the viewing angle θ using (4) and (5), followed by the polarisation parameters ρ and φ with (2) and (3). The polarisation images for diffuse and specular surfaces I diffuse pol and I specular pol are finally recovered using the calculated polarisation parameters with (1).
Depth to correlation (A1) Indirect ToF measures the correlation between a known emitted signal and the received signal. The emitted signal at frequency f M is a sinusoid: and the signal, reflected by the scene, is of the form [30] f where the τ is the time delay between the emitted signal g(t) and the reflected signal f (t). The i-ToF measurement c(x) is the correlation between the two signals: where we only consider the direct reflection signal and ignore the multipath interference (MPI) and sensor imperfections. We are interested in the phase ϕ, proportional to the depth d between the objects in the scene and the sensor: where c is the speed of light and [2π] represents the 2π-ambiguity. Using the four bucket strategy [36] to sample can be obtained to recover the phase ϕ, the amplitude α and the intensity β.
Hence, from a given depth d, one can compute the phase ϕ using (9) and then reformulate the recovered i-ToF correlation using (10), (11) and (12) in turn to form I corr .
Stereo loss L stereo This loss requires that left and right image pairs are accessible during training. While only the left image I l is fed to the network, the right image I r can guide the model towards generating valid depth, and vice versa. More formally, let K l and K r be camera matrices with intrinsic parameters for left and right images respectively, and D a depth map on the left reference frame. Let T left→right denote the transformation that moves 3D points from the left coordinate system to the right. An image coordinate transformed from left coordinate p l to the right image is A backward differentiable warping [31] is used to reproject an image onto the left view as I right→left .
We form a stereo loss L stereo , and related mask loss L mask similarly to [22], which aid network training and deal with occluded pixels respectively as (15) where the photometric error is similar to [22]: Analytical losses L corr and L corr−→pol Depth D corr is firstly inferred directly from i-ToF correlation input, and then two recovered images I corr and I pol are formed. Recovered images represent the 'ideal' input for each modality, i-ToF and polarisation respectively, conditioned on the inferred depth. Since I pol is generated from D corr , we reproject it using D pol to form a recovered final polarisation image I i corr −→ D pol pol , i ∈ {diffuse, specular}. In each case, discrepancies between the recovered image and the true input image provide a strong indication of the quality of the generated depth.
We use this signal to guide the network. Formally where I l is the left polarisation image, I corr −→ D pol pol the recovered polarisation image aligned to I l , I corr the i-ToF correlation input, and I corr the recovered correlation image. We use a min operator for L corr→pol to lift the problem of classifying a pixel as diffuse or specular by computing both possibilities and letting the network select the best solution.
Finally, following [59], we use an additional loss L struct in the objective function, derived from structured-light information (see appendix for further detail).
In summary, depending on the input modalities available at training time, we can add or remove the introduced losses L corr , L corr→pol , L stereo and L struct as appropriate. We explicitly note that hyper parameter tuning for balancing of these loss terms is not required, due to our formulation.
Our total loss L can then be formulated as: where L DF is the L 2 norm between predicted and GT DF.

Data
We next provide details on our custom camera rig (Sec. 4.1) and CroMo dataset (Sec. 4.2), comprising synchronised image sequences capturing multiple modalities, at video-rate across real-world indoor and outdoor scenes.

Camera capture rig
Our prototype custom-camera hardware rig is shown in Fig. 5. Our rig is constructed in order to capture synchronised data across multiple modalities including stereo polarisation, i-ToF correlation, structured-light depth and IMU. We rigidly mount two polarisation cameras (Lucid PHX050S-QC) providing a left-right stereo pair, an i-ToF camera (Lucid HLS003S-001) operating at 25Mhz and a camera (RealSense D435i) for active IR stereo capture. All devices are connected with a hardware synchronisation wire resulting in time-aligned video capture at a frame rate of 10fps. The left polarisation camera is the lead camera which generates the genlock signal and defines the world reference frame. Accurate synchronisation was validated using a flash-light torch and was further confirmed by the respectable quality observed from stereo Block Matching results [29]. The focus of all sensors is set to infinity, the aperture to maximum, and the exposure is manually fixed at the beginning of each capture sequence. The calibration on all four cameras' extrinsics, intrinsics, and distortion coefficients is done with a graph bundle-adjustment for improved multi-view calibration (see appendix for further details).

CroMo dataset
We collect a unique dataset comprising multi-modal captures such that each time point pertains to measurement of (1) Polarisation: raw stereo polarisation cameras produce 2448×2048 px stereo images. (2) i-ToF: 4 channel 640×480 px correlation images. (3) Depth: a structuredlight capture of the scene resulting in a 848×480 px depth image. In addition to the three main sensors, IMU information is recorded to further enable future research directions. Our dataset consists of more than 20k frames, totalling >80k images of indoor and outdoor sequences in challenging conditions, with no constraints on minimum or maximum scene range. We group these sequences into four different scenes which we name: Kitchen, Station, Park and Facades. Despite the multitude of senors, operating ranges are not unlimited and our data collection also does not cover all possible scenarios; we further discuss limitations in our appendix. We report statistics per captured scene in Tab. 1 (lower). These statistics characterise our scene captures and provide useful information, e.g., that the median scene depth differs greatly between indoor (Kitchen) and outdoor (Station, Park and Facades) scenes. This is a strong indicator for whether the i-ToF sensor will perform well. Tab. 1 (upper) provides a comparison with other depth datasets showing that CroMo is the first publicly available, modality rich dataset containing a large quantity of image data.

Experiments
Our experimental design evaluates (1) the effect of multiple modalities, accessible at training time, for monocular depth estimation and (2) the effect that changing network architecture has on depth quality, under consistent input signal. Our capture setup allows us to employ a standard MVS approach [52] on full temporal sequences of polarisation intensity frames (left-camera), to serve as ground-truth depth for our experimental work. This expensive offline optimisation leverages accordances amongst all frames per sequence, affording high quality depth to evaluate our ideas.
Multi-modal training We firstly evaluate combinations of training input signal by changing the number of sensors available to the model. We fix network encoder-decoder backbone components (i.e. ResNet50, analogous to [22]) and train models that leverage cues from a maximum of four sensors; left and right polarisation, i-ToF correlation and structured-light. We show predicted depth improvements, attainable by systematic addition of sensors, and quantify where best gains can be made. The training signal components used for our monocular depth estimation experiments are as follow: Temporal (M) extracts information from video sequences (3 frames), Stereo (S) uses stereo images, i-ToF (T) leverages i-ToF correlations via our two interconnected depth branches (see Sec. 3.2). Finally, Structured-light (L) incorporates an additional mask into the objective function, derived from information provided by our structured-light sensor. The structured-light signal is utilised only when the mask improves the projection loss. We explore alternative strategies to exploit the structured-light signal and discuss details on practical benefits (e.g. convergence speed) in the appendix.
Introduced signal components define our set of training experiments. For example, Stereo and Structure Light (SL) train the model using self-supervised stereo (S) and structured-light (L) information. Experiments therefore use differing subsets of the introduced loss terms (see Tab. 3). Qualitative results are shown in Fig. 6. Unsurprisingly, self-supervised stereo (S) is relatively blurry and struggles to capture fine details, such as the thin, metallic arch on the   Figure 6. Same ResNet50 architecture as in [22] with different modalities: each new modality closes the gap with GT.
of-parameters at inference (MP) and the giga-multiplyaccumulates per second (GMACs) in order to evaluate size and compute-cost per architecture. Architectures consist of the ResNet18 U-Net used in [22] and their supplementary material ResNet50 variant, the p2d architecture [11] using ResNet50 with a different data representation (Stokes), the MiDaS [50] architecture and Ours (see Sec. 3.1, Fig. 3a).
Qualitative results are shown in Fig. 7. It may be observed that the ResNet18 architecture with smallest (MP) fails to obtain good background detail of the swing frame structure (Station sample) or of the tree (Park sample). The ResNet50 variant slightly improves detail, especially with raw measurements instead of Stokes (p2d [11]). Even when increasing network capacity c. three-fold with MiDaS, results are unsatisfying. Our proposed architecture (Ours) requires smaller capacity and computation for a sharper reconstruction of the swing and the tree. We disentangle the benefits of additional sensor modalities from our model contributions, highlighting the advantage of gated convolutions and our DF-based approach towards reducing blur.
Quantitative results are reported in Tab. 2. The smallest architecture ResNet18 [22] performs worst. The larger U-Net ResNet50 performs better, and has been generally adopted [11,21]. Note p2d [11] uses a different data representation (Stokes) for polarisation cf . ResNet50; performance decreases. We believe the Stokes representation, us-

Station sample Park sample
GT Label

Ours
MiDaS [50] p2d [11] ResNet50 [22] ResNet18 [22] Pol. Intensity ing angle directly, is more sensitive to noise and not appropriate for an SSIM loss with the self-supervised stereo (S) training strategy. MiDaS [50] provides second best performance and yet necessitates roughly ×2 GMACs. Our architecture provides best performance while remaining relatively compact which we largely attribute to gated convolutions and displacement field estimation (see Sec. 3.1).

Conclusion
We systematically investigate the effect of using additional information from co-modal sensors at training time, for the task of monocular depth estimation from polarisation imagery. Our exploration is enabled through a unique multimodal video dataset which constitutes synchronized images from binocular polarisation, raw i-ToF and structured-light depth. We quantify the beneficial influence of both passive and active sensors, leveraging self-supervised and crossmodal learning strategies that lead to the proposal of a new method providing sharper and more accurate depth estimation. This is made possible through two physical models that describe the relationships between polarisation and surface normals on one side and correlation measures and scene depth on the other. We believe that our fundamental investigation of modality combination and the CroMo dataset can accelerate research of both spatial and temporal fusion, towards advancing cross-modal computer vision.

CroMo: Supplementary Material
We provide additional materials to supplement our main paper. In Sec. 7.1 we provide observations on the properties of light polarisation. Sec. 7.2 states the specifics for our surface normal estimation process. In Sec. 7.3, we provide additional details for our multi-view camera calibration procedure, Sec. 7.4 provides some further modelling details and finally Sec. 7.5 gives supplementary information on our network architectures and learning parameters. Most natural light sources emit unpolarized light that only becomes polarized if reflected. Hence the type of reflection, illustrated in Fig. S1, either diffuse (d) or specular (s), influences the characteristics of the reflected polarized light. More specifically, the reflective surface influences the relation between the normals' parameters (θ, α) and the polarisation parameters (ρ, φ), defined in Eq. 2 and 3 of the main paper.

Surface normals
In the main manuscript we estimate polarisation intensity using the varying coordinates of surface normals. Hence, the computation of these normals, derived from network depth prediction, plays an important role in the training process. To increase the robustness of estimated normals, we compute the cross products using four distinct pairs of orthogonal directions as in [S4]: The weighted average of these normals is calculated using weights w i where: The final surface normal (unnormalized) is then estimated by their linear combination: Weights w i result in neighbouring pixels of i un that contain strong color disparity, to be down-weighted in the normal computation. This follows from the assumption that such pixels are more likely to represent different objects. Conversely, if neighbouring pixels possess similar color, they are more likely to correspond to the same object and their associated partial derivatives are more likely to provide normals that accurately describe the observed object shape.

Graph-based bundle adjustment
As discussed in Sec. 4.1 of our main paper the calibration of extrinsics, intrinsics and distortion coefficients, for all four capture-rig cameras, is achieved using a graph-based bundle-adjustment [S2] that improves multi-view calibration. We provide here further details of our multi-view calibration procedure.
We start with well established calibration methods [S1] to obtain the intrinsics K k and distortion coefficients d k for each camera C k , where k ∈ {0, 1, 2, 3}. We use a standard pinhole camera model and define C 0 as the left polarisation camera, C 1 the right polarisation camera, C 2 the i-ToF camera, and C 3 the structure light camera. We use five parameters for the distortion coefficients and collect n images of a calibration checkerboard, from all cameras synchronously. In practice we move the checkerboard in front of the cameras while keeping the camera rig stationary. We attempt to cover as wide a field-of-view as possible for all four cameras. We find it is more important to thoroughly cover and account for the extremities of the individual images as opposed to attempting to be visible to all cameras simultaneously. Further, we estimate the rigid transformation for each camera pair composed of C 0 (our world reference), and camera C k in turn, where k ∈ {1, 2, 3}. This provides the extrinsics T k→0 = [R k→0 t k→0 ] for camera C k (with T 0→0 = [I 0]).
These initial intrinsic, extrinsic parameter values and the distortion coefficients are however sub-optimal as they are obtained by solving successive sub-optimisation problems. Towards improving the multi-camera calibration, we define the reprojection error of points X j on the image I i for the camera C k as Where T i 0 is the position of camera C 0 for image i, and x i j is the distorted 2D point from the projection function π(·) which projects a 3D Point X j visible by the camera C k at position T i 0 · T k→0 with distortion coefficients d k and intrinsic parameters K k on image I i . The function dist(·) defines the robustified distance between 2D points, i.e. a Huber m-estimator, and x i j is the 2D point detected on the checkerboard with a corner detector corresponding to the 3D point X j in image I i . The indicator function 1 x i j ∈Ii defines whether the 2D Point x i j is visible in image I i .
Finally, we used a graph-based bundle-adjustment [S2] to model the global problem, for all cameras C k , jointly as: with T 0 0 fixed to [I 0] in order to properly constrain the gauge freedom. All camera calibration parameters are initialised using the values obtained from the original individual calibrations.
This formalism, borrowed from the SLAM community [S3], allows us to optimize all parameters, i.e. the intrinsic, the extrinsic and the distortion parameters for all cameras, jointly. We find the global optimisation process is able to improve our calibration RMSE by ∼5-10%.

Additional modelling details 7.4.1 Reflection ambiguities
A diffuse-specular ambiguity initially exists in our formulation; pertaining to diffuse or specular reflection (see Eq.3, main paper). This ambiguity is addressed during training via the min operator found in Eq.18. We propose to resolve reflection ambiguities (per-pixel) by minimization of the SSIM loss between respective {specular, diffuse} images and the input image, towards consistently providing a valid training signal. Secondly, the azimuthal π-ambiguity is directly accounted for by the formulation of Eq.1; the inherent cos(·) modulation nullifies ambiguity found in its input (2φ component of the argument) and thus supervision is not adversely affected due to φ being modulo π.

Wrappings ambiguities
An analytical solution exists for the correlation to depth transform and a wrapping ambiguity remains. However we highlight that a reconstructed depth, although "phase wrapped", is still able to provide reliable surface normals that can be used to produce (1) the degree of linear polarisation and (2) the Angle of Polarisation, for both diffuse and specular surfaces. Once these are projected to the N 2 referential, this information is used in conjunction with the brightness of the left polarisation image to render valid "Recovered polarisation" images, (see Fig 3b of our main paper).

Polarization intensity recovery
To render the intensity, we require the brightness of each pixel (i un in Eq.1). We obtain the brightness of the polarisation image by channel-wise summing of the left polarisation input pixel values. Two images are rendered following Eq.1; for both the cases of diffuse and specular images. We use a binary mask to select values, pixel-wise, from either the specular or diffuse image. The mask selects pixels such that the minimum SSIM loss between the {specular, dif-fuse} image and the input image are retained.
The two images formed therefore constitute only an intermediary step towards producing a final image. We use a binary mask to then select values, pixel-wise, from either the specular or diffuse image to form a new image containing the pixels that retain the minimum SSIM loss between the {specular, diffuse} image and the input image (i.e. the min in Eq.18 is per pixel). We thus form a final image that contains both specular and diffuse components.

Correlation image rendering from depth
Analogous to the Polarization image strategy, we use the input correlation image (obtaining α, β estimates), in addition to depth information, to estimate both the ambient β and reflectance α, for correlation reconstruction. We provide additional description for the network architecture that we propose in order to process the considered input modalities. Instances of this architecture are depicted as edges 'N 1' and 'N 2' in the system design; see Fig 3a of our main paper.
We employ a standard U-Net architecture, similar to our baseline [22], including skip connections. The encoder is based on a 'Resnet' [28] style block, with the original convolutional layer replaced by gated convolution [63]. The size of the input images are 512×544×12 for polarisation and 640×480×4 for i-ToF, respectively.

Training parameters
To aid reproducibility, we report training parameters and hyperparameters. We use identical training parameters and align with our baseline [22] where possible. We use the Ranger optimiser [S5] and batch sizes of 8, a learning rate of 1e−4 with an exponential learning rate decay. We train all considered methods for 50 epochs.

Comparison with RGB input
A direct comparison with RGB input forms a relevant and interesting line of enquiry. Our custom capture rig does not currently accommodate this modality directly. However, towards investigating this experimentally, we did transform the polarisation input frame to an RGB frame by considering the polarisation intensity of each RGB channel, individually. We note that this is not directly equivalent to an RGB sensor since the bayer pattern differs. We actively decided not to include this experimental work in the main paper to avoid misinterpretation and confusion. Preliminary work evaluating our Polarisation input cf . the noted "Polarisation-converted-to-RGB" showed improvements using Polarisation (RMSE 1.4) over "Polarisationconverted-to-RGB" (RMSE 1.53).

Controlling for capture environment
We note that the i-ToF modality excels in indoor environments, however these represent a relatively smaller portion of our dataset. To corroborate this, we report an experiment that considers our various training strategies, tested on only an indoor environment (Kitchen). The addition of i-ToF (from (S) to (ST)), at training time, significantly improves the predicted depth in this restricted setting (see Tab. S1).  Table S1. Test on Kitchen scene (780 frames): additional sensors can be observed to improve performance. The largest improvement comes from the addition of the i-ToF, (from (S) to (ST)), in an exclusively indoor test setting.

Further analysis of where additional sensors help
We include preliminary further analysis with respect to investigation of scenarios where additional sensors help. We include an example that highlights two points (see Fig. S2).

Additional structured light experiments
The structured light sensor present in our camera rig offers low-noise signal which we find can also be leveraged in a supervised fashion, directly. For completeness, we compare the resulting depth estimation when supervising directly with structure-light (D) and our approach, using unsupervised signals (STLM). The structure-light signal, obtained from our Realsense sensor, is claimed reliable up to a 10 meters range according to the constructor [1]. We thus further investigate by evaluating performance over distinct 0 − 10m and 0 − 20m ranges. Results in Tab. S2 show that the fully supervised method (D) can offer similar performance to our approach (STLM) in the range 0 − 10m yet performance degrades by significant margins when considering the more challenging 0 − 20m range. This highlights the benefits of our unsupervised multi-modal strategy (STLM); leveraging information from multiple sensor sources and an ability to learn to adapt when a particular sensor results in low quality measurement, due to unsuitable physical conditions (e.g. structured light in the 10 − 20m range).

Additional details on the L struct loss
We select to use a structured light loss similar to the loss proposed in [59]. We find that such indirect supervision of  Table S2. Comparison of training strategies for two depth prediction ranges. Our training strategy (STLM) works well in spite of the operational limits of particular sensors.
the structure light signal allows to automatically select the best source of information, particularly in situations where the structure light signal fails or becomes unreliable (as discussed in Sec. 7.5.6). Formally, given a depth from the structured light D struct , the loss reads: We make use of an additional L 1 loss between predicted depth and the D struct depth, when L struct is minimal (see Eq.19 in the main manuscript).

Limitations and Societal Impact
Limitations We note distinct limitations that relate to our sensor setup. Active sensors have limited range and areas of operativity e.g. i-ToF often offers weaker performance outdoors, structure light sensors are of limited range, and polarisation sensors sacrifice spacial sampling resolution for spectral sampling resolution. Our multi-modal ideas attempt to combat these limitations indirectly however we remain bound by the physical laws of light.
Additionally, our current hardware setup is operable by a single person, and yet data capture is currently more cumbersome than e.g. use of a modern smartphone. Training data collection, that involves the acquisition of multiple modalities, currently induces a somewhat larger investment of effort over monomodal capture. With the argument being that the cost may then be recouped when assessing monocular inference time performance. Our hardware rig constitutes a research prototype and form factor likely improves as camera evolution results in further reductions in sensor size, weight and cost.
Finally we would note that our current dataset does not yet capture all possible scenarios and represents but a subset of urban scenes where depth estimation can prove valuable. Future capture sessions will look to enrich and widen the recorded capture scenarios, towards increasing the value of the data resource that we provide to the community.
Societal Impact We note that our proposed CroMo dataset was collected by only two human operators in urban environments. While care was taken towards objective scene capture, such collected data may yet reflect the biases of human operators; influencing specific content, scenarios or capture setups. Efforts towards the reduction of bias, introduced by manual human operators, might suggest mounting of the system on automatic vehicles in future. Additional ideas, toward mitigation of the axis of bias relating to manual data capture, can be considered an interesting future research direction.