Why do we want to normalize loss by the number of tokens? For logging purpose, it might be a good idea to view/save average loss, but for backward pass you can simply use the unnormalized loss.
If we are missing something, please share a side by side comparison of the two scenarios (normalized and unnormalized), to make the question clearer.