Deep image prior 'learns' on just one image
I&#x27;m finding it hard to put into words what I find wrong with this paper, but ... here goes nothing.<p>So, the novel thing here is that an encoder-decoder network applied to an image can learn enough from a source image to be useful.  In some ways that&#x27;s obvious, but the effectiveness of it on reconstruction tasks is certainly surprising.<p>I have two problems, though.  One is that I would take the reconstruction results with a grain of salt.  The examples are clearly lab queens, where the occluded regions are not particularly interesting&#x2F;challenging.<p>Two is the conclusion the authors reach.  Somehow the authors go from the novel discovery I describe above, to saying that somehow the architecture of the network is a prior.<p>Well ... I mean, yeah a network&#x27;s architecture _is_ a prior.  But it&#x27;s not actually significant.<p>See, in the dark ages of machine learning we had only fully connected networks.  They sucked.  They&#x27;d always overfit and underperform or were impossible to train.  Then we finally got convolutional networks, and suddenly a whole slew of machine learning problems became easier and that hurdled us into the current renaissance.<p>But, you see, convolutional networks weren&#x27;t the _only_ reason for the dawn of this new age.  Rather it was three major things:  1) Convolutional layers, 2) more data, 3) more computing power.<p>Some time after the &quot;discovery&quot; of convolutional layers we found out that, hey, our old fully connected networks actually _do_ work.  If you give them enough data and enough computational power, you can get them to perform as well as state of the art convolution networks.  The great thing about fully connected networks is that they assume nothing.  That means A) you can theoretically get better results and B) you don&#x27;t have to spend time designing an architecture.<p>So we already know that architecture isn&#x27;t ultimately important.  You can have a giant, fully connected network, and it _will_ work, if you feed it enough data and have the computational power necessary to train such a beast.<p>Convolutional layers are just simplifications which make training easier.  They are priors in the sense that we know a fully connected layer in image applications would just devolve into a convolutional layer anyway, so we might as well start with a convolution layer.  That &quot;design&quot; is the prior.  But it&#x27;s not mandatory; the network would still function without that &quot;prior&quot;.<p>So ... I&#x27;m not sure how the authors are taking their research and using it to come to the conclusion that their results are because of some magical property imbued into the network by the &quot;priors&quot; of the architecture.<p>They apparently tried other architectures and got poor results, and so they use that to claim that architecture is the only reason their technique works.<p>That&#x27;s like if you started with ResNet for a classification problem, tried other architectures, saw that they performed worse, and then published a paper saying that Residual Networks somehow embody the fundamental forces of natural images in their architecture, and that&#x27;s why they work.  When the truth is that ResNets aren&#x27;t special, they are just easier to train.<p>Another example from the annals of machine learning history:  time and time again when there is a breakthrough in architectures, it&#x27;s usually followed a few years later by a simplification of the architecture.  For example we started with networks like VGG which are these big, hand crafted architectures.  Slowly over time architectures have become _less_ exotic, instead opting to simply define a basic building block repeated N times.<p>The reason for this is because in the intervening years we gather more training data, better training techniques, and more computational power.  So we can instead use a more homogeneous architecture which has _less_ assumptions (less priors) and at the end of the day we get _better_ results.<p>I&#x27;ll repeat that.  We put _less_ priors into our networks and we get better results.<p>So on the one hand we have _all_ of machine learning history telling us that priors in architectures are _bad_.  On the other hand we have this paper which makes some really weird logical leap from &quot;we tried a few architectures, they were worse, so architecture is _key_ to machine learning and it&#x27;s important because we need good priors built into the architecture.&quot;<p>Anyone remember hand crafted feature vectors?  I do.  Those were priors.  Guess what happened when we got rid of them and used generic networks feeding directly from the raw data?  Oh right, all of modern machine learning...