This workflow sounds similar to the DeepDream approach.
If I remember it correctly, one shortcoming from optimizing all input pixels was that high frequency pattern were created (as might be seen for your use case).
From the Inceptionism blog post 10:

Start with an image full of random noise, then gradually tweak the image towards what the neural net considers a banana (see related work in [1], [2], [3] 1, [4]). By itself, that doesn’t work very well, but it does if we impose a prior constraint that the image should have similar statistics to natural images, such as neighboring pixels needing to be correlated.