I don’t have an optimizer. Everything is coded manually using pytorch, perhaps even using a custom optimizer.
To answer your questions directly, I honestly don’t know if your suggestion works the way I want or not (hence my question), it looks essentially the same to the pseudocode that I suggested. My issue is when i % 100 == 0 is true, how do you know that those operations done by some_function won’t be take into account on the next iteration when loss.backward() is called. The reason I suspect it will affect the backward pass is because its modifying the model (or at least it should be in the context of the cases I find problematic). If I am wrong then your example might be enough! Do you think it solves that issue? i.e. that it does not include the operation as part of the computation graph?
Lets take the simplest example I can think of. We train a mdl for T iterations. After that we perturb the model with some noise that is dependent on the norm of the weights/params. Thus, the noise would be a function of W. However, I don’t actually want the framework/engine to take a derivative wrt to the added noise, I just want to train with some algorithm, then perturb it and then train again for another T. In pseudocode:
train_for_T_iterations(mdl)
mdl = mdl + GaussNoise(0,0.25*mdl.W.norm(2)) # defintion of Op(mdl)
mdl.zero_grads() #hopefully this is the line that avoids that Op(mdl) is included in the backward pass 
train_for_T_iterations(mdl)

does it make sense in the context of this example at least?