Deep image prior (DIP) (Ulyanov et al., 2018), which utilizes a deep convolutional network (ConvNet) structure itself as an image prior, has attracted huge attentions in computer vision community. It empirically shows the effectiveness of ConvNet structure for various image restoration applications. However, why the DIP works so well is still unknown, and why convolution operation is useful for image reconstruction or enhancement is not very clear. In this study, we tackle these questions. The proposed approach is dividing the convolution into "delay-embedding" and "transformation (i.e., encoder-decoder)", and proposing a simple, but essential, image/tensor modeling method which is closely related to dynamical systems and self-similarity. The proposed method named as manifold modeling in embedded space (MMES) is implemented by using a novel denoising-auto-encoder in combination with multi-way delay-embedding transform. In spite of its simplicity, the image/tensor completion, super-resolution, and deconvolution results of MMES are quite similar even competitive to DIP in our extensive experiments, and these results would help us for reinterpreting/characterizing the DIP from a perspective of "low-dimensional patch-manifold prior".
In this paper, the authors present a natural image model based on the manifold of image patches.  It is similar to the Deep Image Prior in that it is untrained and has a convolutional-like structure.  It leads to an optimization problem with a reconstruction loss term and an auto encoding term.  The authors show empirical results in time series recovery, non-semantic inpainting, and super resolution.  In the image processing tasks, the performance of the proposed algorithm is on par (sometimes slightly worse, sometimes slightly better) than that of DIP.
I think the perspective of image patch analysis is a useful addition to the knowledge base for unlearned image priors.  That said, the paper says it tackles the questions for why the DIP "works so well" and why convolution operations are "essential for image reconstruction or enhancement".  After reading the paper, it is unclear to me how this work addresses these questions.  In particular, demonstrating a similar convolutional system does not rule out the possibility that there are non-convolutional systems that also explain the effect.  For example, the Deep Geometric Prior paper is nonconvolutional (it is entirely a MLP), and it also has the effect of fitting a smooth signal without training (subject to early stopping).  The DGP could be applied to images as well, resulting in a nonconvolutional deep image prior.  I think the authors should address more clearly and thoroughly the logical connection between their results and the explanation of the DIP, especially in light of the DGP.
The paper claims that the proposed method is more interpretable, and it would be nice if they could demonstrate this interpretability and the benefits it brings in solving image reconstruction problems.  
As a result, I am inclined to recommend a weak reject, but if the concerns above are addressed, I envision my rating could improve upon rebuttal.