If this really causes the problem, my first approach would be to try modifying the respective gradients ‘manually’ (i.e. the embedding gradients taking mean of both gradient by hooking the grad and sub the embedding weights ‘manually’).