Solved by harsh_grover in post #2 
              
              
                I solved it, error was that Pytorch implicitly applies softmax when you use cross entropy and I was applying it again and also the dim =1 part was wrong, since it was a seq output.