Yes, I wouldn’t have wondered if the two functions returned slightly different tensors. But they don’t – only their gradients differ. And you are right, the initial divergence is well in the ballpark of regular float errors. What bugs me is that behind the scenes, the two functions should be doing the same thing; all I am not doing is multiplying the unused vectors in the embedding – which shouldn’t receive gradients anyway.
The random is definitely the same – the number of calls to the random number generator stays the same (embed.weight.size(0)). Even though I’ve changed quite a few things in the code aside from this function, but took extra care not to mess up the random sequence – I even deliberately left a (harmless) bug in the code so that I still get the same numbers.
And you are definitely right, the embeddings diverge by the end, but it is not because the data fed to them is different (that is actually cached, so no randomness is involved there), but because of the cumulative effect of the difference in gradients.
I am wondering if there is a way to see what (symbolic) operations this code translates to. A comparison there might reveal where the difference in gradients comes from.