loneleykid,
Thanks for giving some references. Pyrenn uses directly inverted Jacobian, which can lead to problems for matrix with rank-deficiency. Scipy optimizer is more mature and use trusted region methods and LM implementation as in MINPACK with right factorizations. So using Pytorch for Jacobian and errors evaluation and scipy.optimizer can be an option. Still, not as convenient as a hypothetical direct implementation in PyTorch.