Thank you K. Frank for the good suggestions!
I guess i can use finite differences for gradient approximation at the cost of longer runtimes, but it’s perfect for quickly validating my idea!
The optimization problem can theoretically be solved in torch, but that requires rewriting LOTS of code for just a simple experiment