Solved by ptrblck in post #2 
              
              
                As described in the note for F.conv1d cudnn might non-deterministically pick a kernel or the kernel itself might yield non-deterministic results, which is not the case for the pooling layer.