Hi
I am using LSTM to deal with sequences (sequence to sequence model). In my case the whole training set contains about 7000 sequences with variable length, so I use dataset and dataloader classes to feed my data into the model, with batch_size equal to one.
Now the problem I have is, the processing time is not equal for each batch, it is gradually increasing, for example, in the beginning, processing 50 sequences neededonly 1 minute but after 2000 sequences, processing 50 sequences took about half hour, and the time keeps increasing now, I don’t know what causes this problem? I also noticed that the memory consumption was also increasing, which should not happen. Any ideas?
Thanks!