The second time i run the code, it fails with below information:
Train loss 5454 0.626859 Grad Norm 0.503917 3.96s/it
Train loss 5455 0.583127 Grad Norm 0.679305 3.14s/it
Train loss 5456 0.622990 Grad Norm 0.589986 4.00s/it
Train loss 5457 0.644452 Grad Norm 0.451847 6.29s/it
terminate called after throwing an instance of ‘c10::Error’
what():  get_stat_type_for_pool: invalid pool (get_stat_type_for_pool at /pytorch/c10/cuda/CUDACachingAllocator.cpp:628)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7f9cda062536 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1:  + 0x14214 (0x7f9ce22f7214 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x14a (0x7f9ce22f995a in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::TensorImpl::release_resources() + 0x4d (0x7f9cda052abd in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #4:  + 0x2d9973a (0x7f9cb959573a in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #5:  + 0x2d9a954 (0x7f9cb9596954 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #6: torch::autograd::Engine::evaluate_function(std::shared_ptrtorch::autograd::GraphTask&, torch::autograd::Node*, torch::autograd::InputBuffer&) + 0xf5b (0x7f9cb958276b in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #7: torch::autograd::Engine::thread_main(std::shared_ptrtorch::autograd::GraphTask const&, bool) + 0x3d2 (0x7f9cb9583ce2 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #8: torch::autograd::Engine::thread_init(int) + 0x39 (0x7f9cb957c359 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #9: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f9ce2c354d8 in /home/server/workspace/projects/tacotron2/venv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #10:  + 0xbd6df (0x7f9ce3d066df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #11:  + 0x76db (0x7f9ce885d6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #12: clone + 0x3f (0x7f9ce8b9688f in /lib/x86_64-linux-gnu/libc.so.6)
[1]+  killed               CUDA_LAUNCH_BLOCKING=1 python3 train.py --output_directory /opt/checkpoints/test/ --log_directory log