Thank you for the response @andreiliphd.
In any case, I still want to try to solve this issue.
@ptrblck I used torch.set_printoptions(precision=8). At first glance it appears the weights are the same after being converted from torch to numpy. But if I print the norm of each weight tensor in both formats (numpy and torch) I get small differences such as:
NP : 13.187959
Torch: tensor(13.18795586)
These are definitely not equivalent. This seemed fishy to me so I printed using precision 32 for numpy and torch and I got this:
NP: 0.071544915
Torch: 0.07154491543769836425781250000000
So the Torch tensor appears to have a lot of garbage which was not in the original Numpy vector. Both types are float32 so the Torch tensor should not even be able to hold so many digits (I think).
It’s all pretty weird, and using this checkpoint, the outputs of each layer in the model are slightly different. These differences propagate into a very different final performance.
Am I missing something here? Would really appreciate any help you can spare.
Thanks a lot,
Rodrigo
Edit: My code is basically just:
torch_weight = torch.from_numpy(numpy_weight)