I assume your code works for some iterations and throws an OOM error after a while or is your code running out of memory in the first iterations for the accumulated gradients?
Do you also see the memory leak using the repo code?
Are you accumulating the gradients for each different width?
If you don’t accumulate the gradients, do you still see the memory leak?