As an running example, I want to run SGD until T iterations and then after that I want to continue training but perhaps apply some operation Op(mdl) on the mdl first and then continue training.How do I make sure Op is not registered as an operation to take the derivative?
Would something as follow work:
y=3*x
y.backward()
y = Op(y) # do NOT include in computation graph
y.zeroes() #hopefully this is the line that does the trick
y=4*x
y.backward()

the real pseudo-code:
train_for_T_iterations(mdl)
Op(mdl)
mdl.zero_grads() #hopefully this is the line that does the trick
train_for_T_iterations(mdl)

for something trivial like Op(z) = z + val, if val was a function of the model weights (say gaussian noise sampled from the norm of the current weights), I was concerned that it would take the derivative of val instead of just considering val to be a constant. If its a constant I guess the derivative is zero and we wouldn’t have to worry, but how can I make sure it actually works as I want it to work? i.e. not include some Op in the computation graph where derivatives are being computed.