I too am fascinated by this problem... Your adversarial proposal is very interesting. Let me try to confront the fundamental problem directly.<p>Even if you&#x27;re given a perfect probability distribution over the space of images the solution wasn&#x27;t obvious for me, mostly because we&#x27;re used to thinking of a &quot;best estimate&quot;.<p>The first thing you think of is giving the least-squares estimate (the average), but MMSE exhibits the problem shown.<p>So you might instead try a maximum likelihood estimate; but this too has problems: imagine every car is a sightly different shade of blue (none are quite the same, maybe the manufacturing is unreliable), except 1 in a million cars are red, but the red is very consistent. The ML estimate will pick the red car, which of course is unrealistic.<p>The optimal solution is simply drawing from the underlying distribution, instead of relying on a deterministic &quot;best estimate&quot;: an outside observer won&#x27;t be able to distinguish your generated samples from the true distribution. That&#x27;s why the &quot;Adversarial discriminator&quot; should work.<p>I wonder if there exists a cost function that directly promotes sampling from the underlying distribution without needing the adversarial approach...