Hi,
The issue is that old versions of pytorch were not considering the optimizer step as an inplace operation properly. And so this check was not working properly and was not raising an error even though it was computing wrong gradients. This has been fixed in latest master and this error is expected in this case.
The problem being that you need the value of the parameters to compute the backward but optimizer.step() modifies them inplace. So you will need to wait for all the backward to be done before doing the step. Or you need to redo the forward after the step to use the new value of the weights.