Hi Chris,
Thanks for the reply. I used the global max pool because it reduced my dimension from N x T x M to N x M.  I tried the solution which you proposed but seems like the same thing is happening as it was happening before.