@ptrblck
Thank you for reading long question and reply me.
So you mean that during training, compute loss for every patches and then do mean(all losses).
After that, during validating, compute loss and make  to classify whether the image is defect or not.
Am I following you?