Hi L!



 L-Reichardt:

How can I avoid alpha being trained during .step(), and only train it every nth Batch?


There are a number of ways to do this.  I would probably use two
separate optimizers:
optModel = torch.optim.SGD (myModel.parameters(), lr = 0.1)
optAlpha = torch.optim.SGD ([alpha], lr = 0.1)

(This assumes that alpha is not one of myModel's parameters.)
Now you can call optModel.step() for every batch and only call
optAlpha after every n batches.
Note, that every call to loss.backward() will accumulate loss’s
gradient with respect to alpha into alpha.grad.  If you don’t want
this you might do something like:
optAlpha.zero_grad()
# calculate loss for just one batch ...
loss.backward()   # alpha.grad is now just from one batch
optAlpha.step()



for param in model.parameters():
    param.requires_grad = False
model.alpha.required_grad = True



You can make such a scheme work, but note that if your optimizer is
using momentum or weight decay, having a parameter’s grad be zero
will not prevent that parameter’s value being changed by an opt.step()
update.
Best.
K. Frank