Thank you so much for your reply! It’s quite funny because I realized the same thing yesterday. With too many images there will be too much of a deviation between colors, so the loss will always be pushed away from convergence. A few, preferably similar images will get the loss close enough to 0 to generalize any image into a certain set of colours. Am I thinking the right way :D?
Anyways, I used a single image instead of 20-50 to test it out and it seems that now my network is recognizing shapes and trying to colour other images with the style of that single image, which made me really happy.
I also got rid of the linear layers for now, I am testing it on Conv layers only with identity blocks (I read up on them yesterday). I am getting promising partial results from just those Conv layers:

The loss is already between 0.003 and 0.009  (this is a short test run just to show current loss range, I usually run it for 100,000 epochs to see the decrease) :

I will do any necessary tweaks after I get this puppy closer to 0.
Nevertheless, thank you so much for your offer <3.