Your problem happens when indexes ([C_idx][H_idx][W_idx])  are not unique across cuda grid, i.e. an algorithm includes some reduction. Non-determinism comes from cuda thread scheduler.
Obvious fix is to have unique write indexes per kernel launch. For example, if reduction is only across batch dimension, you can process batch elements one by one (synchronizing between kernel launches) - this is slower obviously. Or write to big “buffer”, so that you don’t have many-to-one index mappings.
ps atomicAdd(double*) requires compute capability 6.0, there is a workaround with casting to (long long)
pps I wouldn’t suggest doing any of this without cuda c++ experience, if you can avoid it