@rasbt @ptrblck I have set:
torch.manual_seed(42)
torch.backends.cudnn.deterministic = True

for both networks on my machine and the work machine. Also, I saved the network state in the working model and loaded it into not working model. Also, I used only 16 same images this time in both models without shuffling.
Here is the output from the working model:

This is the output from the other model:

image.png733×82 8.67 KB

Here is the output of the detect_anomaly, however, the exception is different sometimes:

image.png1875×608 87.1 KB