It looks like Variable was the right call here.
The normalization values of the images were preferred in another tutorial, which had used the same images. Also I switched to Adam, it gives a slight improvement… But still, my loss is heavily obsoleting… Even if it goes down: