Have encountered the same problem, cannot understand why does this happen. If @Soumith_Chintala or somebody from the community could help, would be great!
I have also tried weight decay & dropout, considering the problem as a case of over-fitting, yet the problem persists.