For future visitors- this might be happening because transformer model sharing makes use of git lfs system. So the pytorch_model.bin, that got downloaded is just a pointer.
As @Anubhav1107 pointed out, you can download the file instead of pointer, and you should not get this error.