I noticed this difference too. Whereas my training works fine in Caffe, in PyTorch if I change the learning rate for the same stages/iterations that caffe changes them (step-wise) suddenly I get nan loss values.
Looking at the difference here 6 I thought I can change the adjust_learning_rate to also change the momentum like:
def adjust_learning_rate(optimizer, lr, momentum):
        for param_group in optimizer.param_groups:
            param_group['lr'] = lr
            param_group['momentum'] = momentum / lr

But still, as soon as lr is changed, the loss becomes nan
@Nick_Young how did you solve the problem with SGD discrepancy with Caffe?