Solved by ptrblck in post #6 
              
              
                I guess the “reduction” in the temporal dimension is missing in your model or alternatively you could only use the last time step from the LSTM output. Currently your model is returning logits for a sequence, which doesn’t fit the target.