I see…, so you basically want to separate the gradient from different tokens and minimize them respectively.
I think you can just take the gradient with respect to the output of the embedding layer then minimize it?