A silly question, but are you reading the THPFunction_apply, <omitting python frames>, at::Tensor::mul backward? from frame #24 to frame #14?
Also, one thing I tried was running the code with the batch size as the same size as N, so it would “trick” the backward pass into work and I tracked the size of the Tensor during this. This is what I got
(1st order) U, S, V, G:  torch.Size([1, 4, 4]) torch.Size([1, 4]) torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) Abar:  torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) U, S, V, G:  torch.Size([1, 4, 4]) torch.Size([1, 4]) torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) Abar:  torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) U, S, V, G:  torch.Size([1, 4, 4]) torch.Size([1, 4]) torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) Abar:  torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) U, S, V, G:  torch.Size([1, 4, 4]) torch.Size([1, 4]) torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) Abar:  torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(2nd order) Abar, DetBarBar:  torch.Size([1, 4, 4]) torch.Size([1, 4, 4])
(1st order) U, S, V, G:  torch.Size([4, 4, 4]) torch.Size([4, 4]) torch.Size([4, 4, 4]) torch.Size([4, 4, 4])
(1st order) Abar:  torch.Size([4, 4, 4])

where U, S, V are the elements from the torch.svd and G = torch.diag_embed(gamma(S)). The (1st order) and (2nd order) correspond to CustomDeterminantBackward.forward() and CustomDeterminantBackward.backward(). Weirdly enough, it seems that on the 4th time calling CustomDeterminantBackward.backward() messes up what’s seen in CustomDeterminantBackward.forward(), but only on the final item in the batch. This also seems to happen in CustomDeterminant.backward() too.