Solved by tom in post #3 
              
              
                Good catch with the Upsample.  
I think there currently is no-one systematically investing in determinism. 
(For most cases the problem is much less severe because of a continuous dependence on gradients, but the “fast gradient sign” is very discontinuous by design.) 
Best regards 
Thomas