Solved by Abhilash_Srivastava in post #2 
              
              
                Should you be using a nn.Linear layer here instead of sigmoid activation? 
I see that CrossEntropyLoss being used. So, simply using the logit outputs from the linear should be fine.