I see what you are saying. I am aware that different occurrences of the same token does not result in the same gradient. Lets say I have a sentence abca, then the gradient of a at different positions would have different values. And the gradient of token a at embedding layer Wa=Wa1 + Wa2, where Wa1 is the gradient of the first a token w.r.t the embedding layer and similarly for Wa2 is the gradient of the second a token.
You are saying that I can basically do sum(norm(Wa) + norm(Wb) + norm(Wc)) or something similar and minimize that. For the sake of simplicity, I will define norm() as sum(abs(Wa)**2). You can see why norm(Wa) would be different from norm(Wa1) + norm(Wa2). For this specific application, I thought normalizing them separately is more correct. So that’s why I want to have sum(norm(Wa1) + norm(Wa2) + norm(Wb) + norm(Wc)).