Hello again!
I’ve tried my best to get the least amount of lines possible to reproduce the crash and I think it’s a bit too large to copy and paste within this box. So, I’ve uploaded it to GitHub (CustomDeterminant/pytorch_custom_determinant_example.py at main · AlphaBetaGamma96/CustomDeterminant · GitHub 1)
The reason I can’t write it down to around 50 lines (or so), is because within my derivation of the backward and DoubleBackward, I use some custom functions to get the required matrices I need.
Because I use a Single-Value Decomposition, there’s a chance that I could get a diagonal element within my diagonal matrix, Sigma, near 0 which when inverted overflows and/or returns a NaN. So, I wrote some custom functions to circumvent that. The rest is the definition of the custom function (along with backward, and DoubleBackward) and a nn.Module for PyTorch’s, and my own, determinant (which could probably be done in 1 line now that I think of it). I’ve also written out the shapes of the Tensors as they propagate through the custom functions and network so it’s clear what each lines doing!
(A stack trace of the error is attached below)
Traceback (most recent call last):
  File "pytorch_custom_determinant_example.py", line 195, in <module>
    loss.backward()                 #backprop
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/user/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: The size of tensor a (25) must match the size of tensor b (4) at non-singleton dimension 2
Exception raised from infer_size at /opt/conda/conda-bld/pytorch_1595629427478/work/aten/src/ATen/ExpandUtils.cpp:24 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f01b59e377d in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: at::infer_size(c10::ArrayRef<long>, c10::ArrayRef<long>) + 0x4b8 (0x7f01ec8fba28 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: at::TensorIterator::compute_shape(at::TensorIteratorConfig const&) + 0x10c (0x7f01ecd1de6c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: at::TensorIterator::build(at::TensorIteratorConfig&) + 0x55 (0x7f01ecd1f435 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: at::TensorIterator::TensorIterator(at::TensorIteratorConfig&) + 0xdd (0x7f01ecd1fabd in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: at::TensorIterator::binary_op(at::Tensor&, at::Tensor const&, at::Tensor const&, bool) + 0x14a (0x7f01ecd1fc6a in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: at::native::mul(at::Tensor const&, at::Tensor const&) + 0x47 (0x7f01eca5d317 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xe543d9 (0x7f01ecf923d9 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x7b1990 (0x7f01ec8ef990 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const + 0xbc (0x7f01ed0d7c7c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::mul(at::Tensor const&, at::Tensor const&) + 0x4b (0x7f01ed028c8b in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2c71dc8 (0x7f01eedafdc8 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x7b1990 (0x7f01ec8ef990 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #13: at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const + 0xbc (0x7f01ed0d7c7c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #14: at::Tensor::mul(at::Tensor const&) const + 0x4b (0x7f01ed1be1cb in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x1bc737 (0x7f01f31c4737 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #16: <unknown function> + 0x1bcf66 (0x7f01f31c4f66 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #24: THPFunction_apply(_object*, _object*) + 0x8b5 (0x7f01f353c6f5 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #34: torch::autograd::PyNode::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x183 (0x7f01f353b033 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #35: <unknown function> + 0x30d1017 (0x7f01ef20f017 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #36: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7f01ef20a860 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #37: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f01ef20b401 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #38: torch::autograd::Engine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x37c (0x7f01ef208b1c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #39: torch::autograd::python::PythonEngine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x3c (0x7f01f3532dcc in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #40: torch::autograd::Engine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0x803 (0x7f01ef207e53 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #41: torch::autograd::python::PythonEngine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0x4e (0x7f01f3532bbe in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #42: THPEngine_run_backward(THPEngine*, _object*, _object*) + 0xa29 (0x7f01f3533889 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #60: __libc_start_main + 0xe7 (0x7f020ff2ebf7 in /lib/x86_64-linux-gnu/libc.so.6)

Thank you for all the help!