Hi Pytorch,
I’m trying to train a sequence to sequence VAE, and I keep running out of memory at strange times. Is there any profiling or could there be a reason for this? My sense is that the amount of memory used would scale linearly with the length of the sequence (I have a batch size of one). After i generate my new sequence S’ (of size n x d), I create a pairwise distance matrix of size n x n, but after a few minibatches I run out of memory doing this. I have 16GB of VRAM, but whats more, It routinely seems to fail on sequences that are shorter than previous sequences which ran successfully. Going over my logs it did not run out of memory on a sequence of 1632x1632 but did just fail on a sequence of 1504x1504. What, if anything, could be going on here? I’m happy to provide more details of my implementation if that would help, but it is a straightforward VAE whose encoders and decoders are both based on LSTMs.
Is there a best practice for this sort of thing, managing GPU memory? I guess it’s leaking GPU memory somehow, or something else strange is afoot. Is anyone aware of a profiling tool for this sort of thing? Or am I making a mistake elsewhere?