Thanks for the pointer! After setting ulimit -Sn 500000, now there seems to be a problem with writing and running the tmp files when I run my code with torch.autograd.graph.reset_saved_tensors_default_hooks() after the for loop:
terminate called after throwing an instance of 'c10::Error'
 what():  [enforce fail at inline_container.cc:300] . unexpected pos 320 vs 252
frame #0: c10::ThrowEnforceNotMet(char const*, int, char const*, std::string const&, void const*) + 0x47 (0x7f627ac4e4b7 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x25844b0 (0x7f62c2c164b0 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x257fa8c (0x7f62c2c11a8c in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: caffe2::serialize::PyTorchStreamWriter::writeRecord(std::string const&, void const*, unsigned long, bool) + 0xb5 (0x7f62c2c196f5 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: caffe2::serialize::PyTorchStreamWriter::writeEndOfFile() + 0x173 (0x7f62c2c199e3 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: caffe2::serialize::PyTorchStreamWriter::~PyTorchStreamWriter() + 0x125 (0x7f62c2c19c55 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: <unknown function> + 0xb1cec3 (0x7f62d5797ec3 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x558988 (0x7f62d51d3988 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x559c8e (0x7f62d51d4c8e in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #12: <unknown function> + 0x5548f5 (0x7f62d51cf8f5 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #13: <unknown function> + 0xaa175 (0x7f62d65c0175 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #14: <unknown function> + 0xfbf034 (0x7f62c1651034 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x37083df (0x7f62c3d9a3df in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #16: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7f62d51cab66 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #17: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x132 (0x7f62c3d98672 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7f62c3d90589 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #19: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x53 (0x7f62d5729163 in /home/craig/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #20: <unknown function> + 0xd6de4 (0x7f62d65ecde4 in /lib/x86_64-linux-gnu/libstdc++.so.6)
frame #21: <unknown function> + 0x9609 (0x7f62e658b609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #22: clone + 0x43 (0x7f62e64b2293 in /lib/x86_64-linux-gnu/libc.so.6)

Do you have any idea what the cause might be?