Also removing non linearities (like ReLU) typically makes models easier to optimize, but less powerful. If you only have linear functions (or more precisely affine) like nn.Linear, nn.Conv2d, and + than you can only learn a linear (affine) function of your inputs.
My guess is that there’s some difference in the interpretation of optimizer parameters (like momentum) plus maybe some other small differences. I remember that there’s a different interpretation of momentum in PyTorch vs. Caffe2, but I don’t remember how it differs from Caffe2.
For SGD, I think dampening=True more closely matches the Caffe2 behavior:
https://pytorch.org/docs/stable/optim.html#torch.optim.SGD 1