Thank you soooo much ptrblck.  Now I am more comfortable on my unittest results.
Just out of curiosity, why does the ordering of operations cause this difference? I.E. something to do with cudnn’s algorithm of matrix computation?  I am just curios. I have zero experience with cudnn coding.