I don’t think reproducibility in terms of “exact same floating point number” is reasonably achievable, e.g. AVX vs. non-AVX will make a difference.
If your reproducibility depends on floating point accuracy divergences (sadly a lot of DL stuff does), something bad is going on, anyways.
Best regards
Thomas
P.S.: This isn’t a criticism of your work, but my thought regarding the general situation. In numerical analysis the goal is to get stable algorithms (a nontrivial problem e.g. for stiff ODE problems) to avoid blowup of numerical errors.