If I’m not mistaken, the gradient of the sign method should be zero everywhere.
Since the authors implemented a custom backward method, you could have a look at this tutorial 2 to see, how to do the same in PyTorch.
I’m not exactly sure, what env.step() is doing, but generally if you leave PyTorch and use e.g. numpy, you will detach the operations from the computation graph, which seems to be the case here.