I use the code below to optimize the parameters both in the model and loss.
"trainable = [{‘params’: loss.parameters(), ‘lr’: args.lr},
{‘params’: model.parameters(), ‘lr’: args.lr}] "
and I found a strange phenomenon , if i use the above optimizing method , for training several epoches, the loss descend from 11.7 to 10.8 and just stay there .
but if I put all the parameters in the model ,and use the following code . the loss descend from 11.7 to 1.3
"trainable = [{‘params’: model.parameters(), ‘lr’: args.lr}] "
ps: the weights in the loss using the normalize, because if i didn’t use normalize ,the loss sometimes become “nan”