In the application of machine learning to remote sensing, labeled data is often scarce or expensive, which impedes the training of powerful models like deep convolutional neural networks. Although unlabeled data is abundant, recent selfsupervised learning approaches are ill-suited to the remote sensing domain. In addition, most remote sensing applications currently use only a small subset of the multi-sensor, multi-channel information available, motivating the need for fused multi-sensor representations. We propose a new self-supervised training objective, Contrastive Sensor Fusion, which exploits coterminous data from multiple sources to learn useful representations of every possible combination of those sources. This method uses information common across multiple sensors and bands by training a single model to produce a representation that remains similar when any subset of its input channels is used. Using a dataset of 47 million unlabeled coterminous image triplets, we train an encoder to produce semantically meaningful representations from any possible combination of channels from the input sensors. These representations outperform fully supervised ImageNet weights on a remote sensing classification task and improve as more sensors are fused. Our code is available at https://storage.cloud.google.com/public-published-datasets/csf_code.zip.
The paper proposes an unsupervised method to embed (remotely sensed) image patches such that they are clustered by high-level "image content" rather than by low-level pixel statistics. The argument is that in remote sensing one often has access only to a subset of all possible images/channels, so it would be useful to get the same embedding for each subset depicting the same scene. The basic idea is to generate, during training, two versions of the same patch by randomly dropping out channels, and to penalise the difference between the two corresponding embeddings, using the InfoNCE loss. The latter ensures that the patch remains "recognisable" among a set of random non-matching patches, so as to avoid learning a trivial embedding that is the same for any input.
The high-level goal, to train an encoder that always gives a similar encoding (and consequently similar downstream results) for a given patch, independent of the sensor channels used, is certainly appealing. It is also true that one would like to do this with unsupervised learning or transfer learning, since labelled data is scarce in remote sensing. Unfortunately, in my understanding the paper does not make much progress towards that goal.
Novelty is limited, the work is a variation of the CMC method. Yes, it splits channels randomly, not by views; and uses a Siamese architecture, not separate encoders; which it can afford to do since there are no viewpoint differences. And it computes the loss at more than one layer - which is a fairly trivial and popular thing to do. Moreover it is not well justified in this case, since all that matters in a feed-forward architecture is that the final embeddings are similar. Constraining lower layers might empirically help - in particular speed up learning - but conceptually there is no reason to reduce the lower layers' capacity to somehow turn different inputs into the same output. If they are really very different (think RADAR images) this could in principle even hurt. 
Both in the discussion and in the experiments, I miss crucial and obvious baselines. First, it does in my view not make sense to use pre-trained ImageNet features as the only baseline. Yes, they are known to work surprisingly well also for remote sensing data - but, unsurprisingly, not nearly as well as pre-training on similar remote sensing images. A more natural baseline would be to pre-train on the same type of imagery, with a proxy task for which it is easy to get labels automatically  - actually the paper implicitly suggests that labels from OpenStreetMap might be a good candidate. Beyond reducing the domain gap, this also has the advantage that one is not limited to 3 channels, and that one can use an architecture suitable for the task (e.g., standard architecture with a lot of pooling are known to not perform well for semantic segmentation of remote sensing data). Labels for supervised learning might in many remote sensing tasks be scarce, but to get any old labels and pre-train a reasonable proxy task is still easy, as map data is abundant.
Perhaps even more important is another natural baseline. In spirit, the proposed method is closer to a generative embedding, since the supervision signal is not a discriminative task, but rather encourages embeddings to stay unique for each patch, and hence in principle fully decodable. Arguably, the most obvious way to achieve the same thing is a Siamese pair of auto-encoders for the two training patches, with an additional loss on the difference between the latent representations. Without that baseline it is impossible to tell whether the InfoNCE loss does a good job - in some sense it feels like making the task harder if one demands not only that a patch of water can be decoded from the embedding, but that it should be distinguishable from any other patch of water that resembles it even in raw pixel space.
The data used in the experiments is somewhat unconvincing, as it is really the opposite of diverse, multi-modal remote sensing channels. The same four R-G-B-NIR channels from sensors with not so different resolutions. The claim that the method homes in on high-level similarity and overcomes low-level radiometric differences would be much more credible if one uses short-wave infrared, thermal, perhaps even Radar images. Moreover, the gains are disappointingly small: even in the best case of 12 channels, the improvement over ImageNet pre-training is roughly from 4.5/10 correct neighbours to 5/10 correct neighbours. The even lower gains with fewer channels underline the need to have a pre-trained baseline without a massive domain gap. The classification accuracy of kNN does improve with the proposed embedding (at least for some combinations - for Pleiades RGB it is the same), but that experiment is unnatural. If one has the labels to do kNN, one might as well use them to train a proper classifier, i.e., add a couple of fully connected layers and fine-tune in a supervised fashion.
A number of technical details in the experiments remain unclear. Section 4.2 suggests the channels in Fig. 4 were added in a fixed order - why? There are many potentially viable combinations of, say, 4 channels, not just one. Furthermore, Fig. 5 suggests to me that the learned similarities are in fact not "high-level", but (up to rotation invariance) fairly low-level texture properties like "homogeneous green patch", "bright straight line on dark background", etc. I am not sure what suggests to the authors that there is much sensor invariance. Again, channels with really different wavelength would be more convincing here.
In terms of presentation, the paper repeatedly makes strong, but unsupported claims about the special properties of remote sensing images.  It makes no sense to state that standard pictures often have "one subject" - ImageNet does, but a look at standard datasets like Cityscapes or DepthInTheWild shows that it is not true in general. In the same vein, I don't see why the content changes more unpredictably at the same location across time - arguably the rate of change is much lower, because only large-scale changes impact the image, and because there aren't many dynamic occluders. And the "distributional hypothesis" certainly also holds for remote sensing images, just at different scale - otherwise people would not use all sorts of smoothness priors, super-pixel segmentations, etc. when working with them. And I do not understand why the adjacent-patch approach of CPC needs nearby patches to be similar - in my view it only requires that nearby patches are not conditionally independent, so basically any type of patterns.
A small detail: some references, while not wrong, seem rather arbitrary. E.g., (Wu 2018) is a little-known random example. Why not use earlier, more standard references - for semantic segmentation of remote sensing data with deep learning one could for example think of (Maggiori, Audebert, Marmanis, Kampffmeyer, Sherrah, ...).
Overall, while the paper brings up a valid question, it does not give a convincing answer. The justification of the method and its novelty is to some degree contrived, the dataset is ill-suited to really prove the point, and the natural baselines are missing.