Sorry, I’ve just added an edit to my reply above. I think it could be the issue with the fact that the elements within laplacian have their batch as 1 and y having its batch as B? That could surely result in an error? If I make y have a batch of 1 like in my edit. It gives the correct shape, but if I put B = 1000 I get a recurssion error which I assume it a different problem entirely and nothing to do with my custom function?
EDIT: I think that might be the problem as A is stored within ctx in the forward method of CustomDeterminant. So the y = Net(x) line will have a stored A of size [B, N, N]  but the A stored within the Net of the laplacian Tensor will have an A of size [1, N, N]. That surely would cause a mismatch in dimensions?