I have fixed the bug. I forgot to add the weight initialization to the output layer in the training code of Net1. It caused this weird training loss trend. When I used torch.manual_seed() and the weight initialization, these three ways are equivalent.
Thank you for your help.