When I have output as  [batch size, vocab size, seq length] and my taregt as [batch size, seq length], the model does not learn. Doing this way is important for me since loss function in turn outputs [batch size, seq length] and then allows me to take average over both timesteps (i.e. seq length) and batch. Any idea why my model does not work in this case?
I tried sqeezing, where my output now is [batch size*seq length, vocab size] and target is [batch size*seq length]. In this approach my model learns but it does not allow me to take average over timesteps.
note: vocab size for me is nothing but the number of classes.