I’m not sure if you really need to compute the mean of the losses of all patches from the same image and I would assume that you could directly use each patch separately.
During validation you would have to use all patches from the image, apply your threshold, and classify the image based on the classification of all patches corresponding to this image.