I’ve just ran it again with the TORCH_SHOW_CPP_STACKTRACES=1  enabled and the stack trace is quite extensive, which is,
RuntimeError: The size of tensor a (25) must match the size of tensor b (4) at non-singleton dimension 2
Exception raised from infer_size at /opt/conda/conda-bld/pytorch_1595629427478/work/aten/src/ATen/ExpandUtils.cpp:24 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7f63bd7bc77d in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: at::infer_size(c10::ArrayRef<long>, c10::ArrayRef<long>) + 0x4b8 (0x7f63f46d4a28 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: at::TensorIterator::compute_shape(at::TensorIteratorConfig const&) + 0x10c (0x7f63f4af6e6c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: at::TensorIterator::build(at::TensorIteratorConfig&) + 0x55 (0x7f63f4af8435 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: at::TensorIterator::TensorIterator(at::TensorIteratorConfig&) + 0xdd (0x7f63f4af8abd in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: at::TensorIterator::binary_op(at::Tensor&, at::Tensor const&, at::Tensor const&, bool) + 0x14a (0x7f63f4af8c6a in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #6: at::native::mul(at::Tensor const&, at::Tensor const&) + 0x47 (0x7f63f4836317 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #7: <unknown function> + 0xe543d9 (0x7f63f4d6b3d9 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0x7b1990 (0x7f63f46c8990 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const + 0xbc (0x7f63f4eb0c7c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::mul(at::Tensor const&, at::Tensor const&) + 0x4b (0x7f63f4e01c8b in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #11: <unknown function> + 0x2c71dc8 (0x7f63f6b88dc8 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #12: <unknown function> + 0x7b1990 (0x7f63f46c8990 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #13: at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const + 0xbc (0x7f63f4eb0c7c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #14: at::Tensor::mul(at::Tensor const&) const + 0x4b (0x7f63f4f971cb in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0x1bc737 (0x7f63faf9d737 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #16: <unknown function> + 0x1bcf66 (0x7f63faf9df66 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #24: THPFunction_apply(_object*, _object*) + 0x8b5 (0x7f63fb3156f5 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #34: torch::autograd::PyNode::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x183 (0x7f63fb314033 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #35: <unknown function> + 0x30d1017 (0x7f63f6fe8017 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #36: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7f63f6fe3860 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #37: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7f63f6fe4401 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #38: torch::autograd::Engine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x37c (0x7f63f6fe1b1c in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #39: torch::autograd::python::PythonEngine::execute_with_graph_task(std::shared_ptr<torch::autograd::GraphTask> const&, std::shared_ptr<torch::autograd::Node>) + 0x3c (0x7f63fb30bdcc in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #40: torch::autograd::Engine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0x803 (0x7f63f6fe0e53 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #41: torch::autograd::python::PythonEngine::execute(std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&, std::vector<at::Tensor, std::allocator<at::Tensor> > const&, bool, bool, std::vector<torch::autograd::Edge, std::allocator<torch::autograd::Edge> > const&) + 0x4e (0x7f63fb30bbbe in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #42: THPEngine_run_backward(THPEngine*, _object*, _object*) + 0xa29 (0x7f63fb30c889 in /home/user/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #60: __libc_start_main + 0xe7 (0x7f6417d47bf7 in /lib/x86_64-linux-gnu/libc.so.6)


So, my loss term contains 1 term which is dependent on the Laplacian of the output (along with some other terms which don’t require any custom autograd functions). So, what I do is I take my batch of inputs (x) of shape, [25, 4] then pass it one-by-one through the function to get the Laplacian. So, along the lines of
lap = torch.zeros(x.shape[0]) #tensor to store laplacian for each batch value
for i, xi in range(x.shape[0]): #iterate over Tensor
  hess = torch.autograd.functional.hessian(Net, xi.unsqueeze(0), create_graph=True) #calculate hessian
  lap[i] = torch.diagonal(hess.view(x.shape[1], x.shape[1])).sum() #sum over diagonal to get the Laplacian.

So, it can do this Laplacian calculation for each input. I then combined with some other loss values, then sum over this to get a single value to represent the loss for a given batch of inputs. However,  when I called loss.backward()  I get the error from above.
RuntimeError: The size of tensor a (2500) must match the size of tensor b (4) at non-singleton dimension 2

Because, my function is a determinant, and my backward is the cofactor matrix (by definition), and the DoubleBackward is the derivative of the cofactor w.r.t the input matrix. So, that reduced sum should get rid of the need for the size of [B, ]?
Unless I messed something within my Backward or DoubleBackward definitions, but the gradients that are returned by those functions are of the same shape as its input within its forward method so it seems the shapes are fine? Granted I could have (and most likely have) done something wrong within there, if the shapes are correct and the calculations are wrong in some way, I would just get the incorrect gradients that would never hit a local minima of my loss function. I shouldn’t get a mismatch error, no?