@DiffEverything in my case I found a super useful trick that allows the GPU RAM to stay quite constant (vs increasing per decoding time step previously).
Basically, I’m running .backward() now at each timestep. backward frees some buffers, that’s what making our model ‘lighter’ but may also cause issues. If your running backward on a variable, each variables that was used to compute it will be freed. Thus, if you need one of this variable in the next iteration it will fail, telling you already called backward. The trick is to explicitely copy variables that will be needed in next step. Doing this you are freeing anything but the vairable you copied.
Quite hacky but make sense and worked well for me. I couldn’t run an iteration with a batchsize=1 on my 8GB GPU, I can now run it with 64 batchsize and not close to be OOM.