I found another thread I made in the past Random core dumps and segmentation fault - #2 by ptrblck 2 when I was unable to reproduce the problem, but as this is a long time after and seems like a totally different issue, I will make a new thread.
I am using pytorch’s higher GitHub - facebookresearch/higher: higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps. for some meta learning and I get a core dump every time I run the code. The code does go to NaN sometimes so there is some other problem I should fix, but this core dump still should not happen I believe.
torch: 1.8.1+cu111
higher: 0.2.1 (I believe)
Any idea what could be causing this?
Thread 32 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffe40a80700 (LWP 1557747)]
0x00007fff94b7d48f in at::cuda::getCurrentCUDABlasHandle() () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
(gdb) bt
#0  0x00007fff94b7d48f in at::cuda::getCurrentCUDABlasHandle() () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#1  0x00007fff94b6b7dd in void at::cuda::blas::gemm<float>(char, char, long, long, long, float, float const*, long, float const*, long, float, float*, long) ()
   from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#2  0x00007fff95d878f0 in at::native::(anonymous namespace)::addmm_out_cuda_impl(at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::Scalar, c10::Scalar) ()
   from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#3  0x00007fff95d89092 in at::native::mm_cuda(at::Tensor const&, at::Tensor const&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#4  0x00007fff94bc2848 in at::(anonymous namespace)::(anonymous namespace)::wrapper_mm(at::Tensor const&, at::Tensor const&) ()
   from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#5  0x00007fff94bc288f in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&), &at::(anonymous namespace)::(anonymous namespace)::wrapper_mm>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&> >, at::Tensor (at::Tensor const&, at::Tensor const&)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#6  0x00007fff8303e596 in at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const [clone .constprop.4808] () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#7  0x00007fff83043bbf in at::mm(at::Tensor const&, at::Tensor const&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#8  0x00007fff8492791c in torch::autograd::VariableType::(anonymous namespace)::mm(at::Tensor const&, at::Tensor const&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#9  0x00007fff84927e8f in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&), &torch::autograd::VariableType::(anonymous namespace)::mm>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&> >, at::Tensor (at::Tensor const&, at::Tensor const&)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#10 0x00007fff834907d6 in at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const [clone .constprop.1487] () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007fff834949ff in at::Tensor::mm(at::Tensor const&) const () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#12 0x00007fff85366911 in torch::autograd::generated::details::mm_mat1_backward(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::Scalar const&) ()
   from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#13 0x00007fff8473c093 in torch::autograd::generated::AddmmBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) ()
   from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#14 0x00007fff84df9771 in torch::autograd::Node::operator()(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007fff84df557b in torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#16 0x00007fff84df619f in torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) () from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#17 0x00007fff84ded979 in torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) ()
   from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#18 0x00007ffff5571293 in torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) ()
   from /home/jeff/.venv/env/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#19 0x00007ffff65c8d84 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#20 0x00007ffff7d9f609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#21 0x00007ffff7edb293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95