Hi,
Sorry for the late reply. I was on holiday. 
Yeah, I was also thinking that maybe its because of the model architecture. As long as its not a code bug, I think I can solve the problem.
Hmm…as for weight initialization, I thought PyTorch used xavier initialization as default?
Thanks.