Solved by zamannafi in post #2 
              
              
                Ok so I figured it out. My learning rate was way too high apparently so setting it to something like 3e-4 fixed the issue. And I was also able to verify that the way I am sending in the input to the decoder is correct.