Before looking into it further, to rule out randomness, have you

used a fixed random seed for weight initialization and minibatch shuffling?
used the same algorithms (e.g., same convolution operation)? I.e., torch.backends.cudnn.deterministic = True


EDIT: Another thing to check

Yes, the  LAE  and the  ELoss  are pretty similar in the first epoch on both machines. I noticed that activations and gradients blow up in the very first batch for no apparent reason,

I’ve seen that before on old PyTorch versions with log_softmax (I think there was a bug fix at some point). Are both machines running PyTorch 1.0?