I have also appropriately initialized it and have  used RELU to prevent the vanishing gradient problem. I also used paremetrized RELU or leaky RELU to make sure ‘dying’ relu problem is not  encountered.
I also have a question. should the labels given to the nn.CrossEntropyLoss be one hot encoded? I dont think thats a necessity.