I went back and looked at the stat_dict of both the locally trained GPU model and the cloud TPU model, and they do have the same precision, around 4 decimal places, so what I said in the original question was incorrect.
After some more reading it sounds like the built in bfloat16 type is part of what makes TPUs so fast, and I don’t understand all the math but I think it can produce the same range of values, so that might not be my issue.  I also should note that I’m using a pytorch_lightning module, although I wouldn’t think that would matter.  I might try calling float() to see if that changes the output, thanks!