That is a neat approach!
A small difference (I seem to see ~1e-7 often when using float32) is expected when you compute the same thing in a different way (that is mathematically equivalent).
Are you sure you have the exact same random for the dataloader and dropout? My guess would be that you have different embeddings by when you get significantly different results.
Best regards
Thomas