Im trying to do some experiments with embeddings and generating images, but have not been able to get a model that generates  good output.I created a very simple model made up of an embedding layer and linear layer  that takes a pair of an image and some text that represents that image and then tries to regenerate that image.  I keep getting random noise as the output of the model instead of an image.  I am testing on just trying to get the model to overfit on a single example ,but I get just get back random garbage.  I train for 200 epochs. Even if the architect is not good, shouldnt the model  be able to overfit and duplicate the exact image? I use mean squared error  on the pixels comparing the training image to generated image, and the image is only 40x40 pixels.  Am I doing something fundamentally wrong? Can give me pointers for this?  Here is the full repo with training data: https://github.com/jtoy/overfit_image_pytorch 105
Here is what the model generates:
https://github.com/jtoy/overfit_image_pytorch/blob/master/results/0_0.png 46
Here is  the training example Im trying to overfit: https://github.com/jtoy/overfit_image_pytorch/blob/master/results/0_0target.png 33