Generative deep learning has sparked a new wave of Super-Resolution (SR) algorithms that enhance single images with impressive aesthetic results, albeit with imaginary details. Multi-frame Super-Resolution (MFSR) offers a more grounded approach to the ill-posed problem, by conditioning on multiple low-resolution views. This is important for satellite monitoring of human impact on the planet -from deforestation, to human rights violations -that depend on reliable imagery. To this end, we present HighRes-net, the first deep learning approach to MFSR that learns its sub-tasks in an end-to-end fashion: (i) co-registration, (ii) fusion, (iii) up-sampling, and (iv) registration-at-the-loss. Co-registration of low-res views is learned implicitly through a reference-frame channel, with no explicit registration mechanism. We learn a global fusion operator that is applied recursively on an arbitrary number of low-res pairs. We introduce a registered loss, by learning to align the SR output to a ground-truth through ShiftNet. We show that by learning deep representations of multiple views, we can super-resolve low-resolution signals and enhance Earth observation data at scale. Our approach recently topped the European Space Agency's MFSR competition on real-world satellite imagery.
This paper proposes an end-to-end multi-frame super-resolution algorithm, that relies on a pair-wise co-registrations and fusing blocks (convolutional residual blocks), embedded in a encoder-decoder network 'HighRes-net' that estimates the super-resolution image. Because the ground truth SR image is typically misaligned with the estimation SR image, the authors proposed to learn the shift with a neural network 'ShiftNet' in a cooperative setting with HighRes-net. The experiments were performed on the ESA challenge on satellite images, showing good results.
Overall, I found this paper interesting, and the method described is both clever and efficient. While some points need to be clarified, I am in favor of accepting this paper to ICLR.
Positive aspects:
- the paper is very clear and easy to read, with nice figures.
- a sensitivity analysis on many different parameters or types of inputs are made, which makes this paper an interesting research paper. For example, the tests on the type of reference image to stack at the beginning are very interesting.
- While I am not an expert on super-resolution, I do see a clever algorithm, that can be for example used with different number of input views. 
- The end-to-end framework is also quite interesting as it allows to be spread easily across the very large satellite images users, with the code aldready publicly available.
- Lastly, the results are good wrt to the state-of-the-art, as the algorithm was proposed during a 2019 challenge and was in the top ones on the private leaderboard.
Remarks and clarifications:
- After looking at the challenge website, it is quite strange to find that the results values in Table 1, in terms of public and private scores, differ from the actual leaderboards, even if the metric is still the same. Thus, I was not able to understand (i) if the authors really participate to the challenge (ii) what method they presented during the challenge as 3 or 4 are shown here (iii) what was their ranking in the real leaderboard. It is important to be precise. From what I see, they might be second on the private score (the one that matters), if so it can be clarified also in the abstract where the word 'topped' was used.
- cPSNR metric : (i) can you please explain the acronym; (ii) it was the competition metric, so it means it is not 'we use' but 'the challenge used' - which will also show that it was not a choice to satisfy your results, but a standard metric.
- Lanczos interpolation: can you please explain in a small sentence in what this interpolation differs from the others?
- ShiftNet: what is this network? We only know tha it is adapted from HomographyNet, but we don't have any information about how it is composed. We just know from App. 1 that it has a very large number of parameters (34M). Why is that? Why a rigid registration needs such a large number of parameters?
- median anchor image: this is one of the interesting points of the paper. Can you please just clarify that you take a naive median image? Such as: for pixel (i,j), med(i,j)= med( LR1(i,j), LR2(i,j) ,...) .
- You are saying 'each imageset is padded to 32 views'. What do you mean? I thought your network was able to be used with different numbers of views. How did you pad your imageset?
- you list 'several baselines': for me, a baseline is something to compare to - usually, it is even something easy, such as the ESA baseline. But in your list, you are mixing baselines, other state-of-the-art methods (some of which you don't beat, so it's not a 'baseline'...), and your methods. It is very difficult to know which ones are yours. Please make different lists and/or identify yours in the table 1.
- What is the Ensemble method? It appears that in the paper, you explain the 'HighRes-net + shiftNet' method, but actually, the 'Ensemble' is the better one, while not clearly described. I don't understand what outputs are averaged - I thought HighRes-net and shiftNet were performed simultaneously.
- How did you select the hyperparameters of your model?
Scientific questions: 
- Is it possible to have views of different sizes as inputs? Or views with missing parts?
- Usually, we have a high resolution image that is different in terms of type of acquisition (like a different type of satellite, but also true in medical images for ex.). Is your network ready for that? I mean: can we constrain the method even if it is not the same type of acquisition as ground truth? In that case, is that possible to have a super-resolution of the type of the input LR images?
Typos: 
- is comprised of -> is composed of
- in Table 7, the bold number should be the beta=infinity as it is the best one. It will be clearer, even if of course, a good train score does not mean a good method because of overfitting.