Thanks Alex for your help and thanks @tom for your detailed answer, I will explore your suggestions. I have never written any CUDA code, but I found the occurrences of atomicAdd in the code. For example:
atomicAdd(&shift_grad_buffer[0][C_idx][H_idx][W_idx], pixel_H_grad);

I tried your enhanced approximate determinism suggestion with the following:
shift_grad_buffer[0][C_idx][H_idx][W_idx] = (float) (shift_grad_buffer[0][C_idx][H_idx][W_idx] + pixel_H_grad); but it seems to do the same as atomicAdd. I used the same notation as in C for a type casting and I added both terms. I am sure that it is quite possible that my solution is not correct. I will try to debug the CUDA code to see what happens. Thanks!