I continually tweaked my network until I was finally able to get deterministic results. I removed the MaxPools first (even though I don’t think this uses atomicAdd), and as expected it changed nothing, but when I then also removed the convTranspose2d’s (being left with only conv2d, BN, ReLU for the entire network), the results were deterministic.
Comparing ReLU reuse in this configuration yields the exact same results. Thus, I can check this issue off as having been statistically random noise/bad luck in terms of the seemingly different results that I received.
Conclusion: Simple stateless ReLUs can be reused without changing anything (whether memory use or the numerical results).