I think I found the problem.
Inside backward, I created a big sparse matrix, and then multiplied it by a vector. So I did something like
big_sparse = big_sparse.cuda()
return torch.mm(big_sparse, dense_vector.view(-1,1))

With some debugging, it appeared that torch.mm 15 was responsible for the huge CUDA leaks. Therefore I tried to use make that operation run on CPU, with
return torch.mm(big_sparse, dense_vector.view(-1,1).cpu()).cuda()
With this change the memory leaks are not there anymore. I don’t know if torch.mm 15 is creating leaks into RAM now, but with a quick check it did not appear to be the case.
I have raised an issue on the github page : https://github.com/pytorch/pytorch/issues/2912 81