How are you comparing these different approaches? Are you seeding the code or reusing the same state_dicts? Is each approach deterministic but yields different results when compared to each other?
In the latter case how large are the differences?