For numerous domains, including for instance earth observation, medical imaging, astrophysics,..., available image and signal datasets often irregular spacetime sampling patterns and large missing data rates. These sampling properties is a critical issue to apply state-of-the-art learning-based (e.g., auto-encoders, CNNs,...) to fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest. In this paper, we address the end-to-end learning of representations of signals, images and image sequences from irregularly-sampled data, i.e. when the training data involved missing data. From an analogy to Bayesian formulation, we consider energy-based representations. Two energy forms are investigated: one derived from auto-encoders and one relating to Gibbs energies. The learning stage of these energy-based representations (or priors) involve a joint interpolation issue, which resorts to solving an energy minimization problem under observation constraints. Using a neural-network-based implementation of the considered energy forms, we can state an end-to-end learning scheme from irregularly-sampled data. We demonstrate the relevance of the proposed representations for different case-studies: namely, multivariate time series, 2 images and image sequences.
The paper proposes a framework to learn representations of signals when they the signals are irregularly sampled.  They propose to do this by using some modified iteration steps from DINEOF algorithm. In addition to this, they propose a new energy function which is inspired by the idea of Gibbs distribution. The parametrize the energy function by some convolutional filters with a constraint. As claimed by the paper in the introduction, they only use the under sampled signals to learn as opposed to using the fully sampled ground truth signals in the previous deep learning basic approaches.
I choose to reject this paper. 1) The paper doesn't justify what these representations are or where these representations will be used 2) There is no theoretical or extensive empirical motivation that the representation that is coming out of this network is actually useful for downstream tasks of interest. 3) Ill-defined evaluation metrics 4) Missing comparisons with basic models - why not compare with in-painting models for images and time series in experiments? 5) The paper is hard to read, not well organized and missing a lot of details. 
Main Argument:
It is not clear to me the motivation for learning these representations or how the usefulness of this representation will be evaluated. The only place where I see a use for representation other than reconstruction of the signal is in Table 1. They use a 3 layered MLP on top of the representation from auto encoder. Why is this interesting? If you're trying to claim that the representation is powerful enough, shouldn't you be able to do the classification with just a linear classifier? Can you justify the usefulness of representation for some downstream tasks in your other experiments? 
In any case, I don't understand why the authors don't compare with a simple auto-encoder. The authors assumes that they know where the signal is sampled and where is is not. Why not train an auto-encoder with MSE loss, but calculating MSE over only those pixels which the authors know are sampled. Comparison with such a model will be helpful in understanding if the framework is adding anything of value.
Authors useful I-Score, R-Score, AE-Score and C-Score to compare across models. These metrics are not defined in the main text (or any reference) but there is a loose (english) definition in the caption of Table 1. For example, R-Score, is the "reconstruction performance of known image areas". Why are these numbers in percentages? Shouldn't you be using MSE, PSNR or SSIM? Why are some of scores negative?
If you are comparing how well you're interpolating for natural images or time series, you should compare with standard interpolation techniques in image processing and time series analysis. 
The paper have a lot of missing details, including simple things like not linking to a Figure or Section the authors are referring to. Some examples are:
1. Authors motivate using Gibbs energy by saying that in auto encoders you have to project to a lower dimensional representation and gibbs energy based solution has no such dimensionality reduction constraint. There are no such constraint in AE as well - you can vey well have your hidden representation to be over complete - so this is not a good motivation for Gibbs energy
2. Main section of 4.1 says that they use MNIST but the caption of Table 1 says they use Fashion MNIST
3. The naming convention of the model is hard to follow and the authors notation itself is inconsistent. 
Some other problems:
1. In Abstract, they say 2. images. Why 2?
2. Introduction, the citation for Gaussian priors is empty
3. Last word before beginning of section to, broken link
4. In the second para of 3.2 did you mean to say solve for (2) instead of (3)?
5.  What is "XXX" matrix in second para of 3.2?