Solved by ishan_pytorch in post #2 
              
              
                Nevermind. I did softmax over the wrong dimension in my predict function. Hate to see it. Interestingly enough, the model still predicted well even when the softmax was along the wrong dimension!