omarfoq:

output = self.rnn(encoded)


You are right, I had some typos in the previous code (I edited when writing the answer, I am sorry for that). But still the problem is that if you run this code in the first time it will give:
1196032
3858432
2179072

Then if you un it a second time it will give:
2179072
3817472
2179072

But I think this is not related to gradients at all, I observe the same thing when running with torch.no_grad(). Apparently there is some part of the memory that is allocated when doing the forward pass, I guess that this is part of memory allocated by the hidden state in the case of RNN, what do you think?
Here is a link if you want to try it directly.