Thanks! Using your idea I was able to drill down to the problem. Using a very small dataset, I could immediately overfit (training accuracy=100%) the model if I don’t use packing – didn’t happen when I used packing, initially. I finally got it to work with packing when I used a batch size of 1.
I’m pretty sure now, that I cannot simply used packing and X=X[-1] to get the last output. When batch size = 1 the GRU output dimension is (seq_len, batch_size, dim) where seq_len is only the length of the sequence without padding. If I have larger batches seq_len is length of the longest sequence in the batch. So when I do X=X[-1] I get meaningless output for all shorter sequences that have padding. I could confirm this by making sure that all my sequences in my mini dataset have no padding. Then I could overfit my model even with packing.
My current solution is there not last output but the final hidden state of the RNN. For this, used the approach outlined here 34. Not 100% sure if this is the (most) correct way, but now I can train my model on the original dataset with packing and get the expected test accuracy of 80%.