Hi Victor, sorry for getting back to you late (it took me some time to install the latest PyTorch onto a machine with CUDA).
In my code that does the actual training (not the example code above), I tried to put torch.autograd.graph.reset_saved_tensors_default_hooks() to after the for loop, but got
...
AttributeError: 'SelfClosingTempFile' object has no attribute 'fp'
...
RuntimeError: OSError: [Errno 24] Too many open files: '/tmp/tmpw5thrpry'

I have no control over how many files the process can open at the same time, though, and I did not find how to control the number of created files in the SelfClosingTempFile() class (like in tempfile.TemporaryFile()). Do you know if there is a workaround? Thanks!