Thank you @colesbury, I posted a reply earlier saying I found torch.randn() produces consistent results but inconsistent initialization parameters inside modules.
Then I realized I was testing different module designs… Indeed everything is consistent across different hardware.
One other question though, previously I used the following line to check for consistent random number yet without specifying device='cuda' I got inconsistent resutls. What do you think is the reason?
print(f'testing torch random seed {torch.randint(1, 99, (5,))}')

Thanks again!