Hi, thank you for your quick reply.
I am using the loss functions borrowed from face recognition modules(like CosFace 34, ArcFace 14, etc).
In those loss function, they will normalize both weights of the last linear layer and features before the matmul operation. (for calculate the cosine values)
If I understand this correctly, I am not trying to initialize those weights, I need both of weights and features are normalized before the matmul operation.
So I think the normalize operation should also be considered when calculating the gradient ?
That’s why I said torch.no_grad() is not suit for me.
Please correct me if I misunderstood something, many thanks.