Same thing is happening with my model. Model is not learning when I have output as [batch size, number_of_classes, seq_length] , and target as [batch size, seq_length]
It is important for me to make sure that I take average over both timesteps and batch while calculating cross entropy loss. Is there any solution for this?