I noticed some big differences in the runtime of Conv1d when using different values for out_channels. Below you can see three tests I ran with different out_channel sizes.
import torch
import timeit

class CNN(torch.nn.Module):
    def __init__(
        self,in_dim, out_dim,kernel_size,
    ):
        super().__init__()
        self.conv_1 = torch.nn.Conv1d(in_dim, out_dim, kernel_size)
    def forward(self, x):
        return self.conv_1(x)

def test_run(x,cnn):
    output = cnn(x).sum()
    output.backward()

device = "cuda:1"
x = torch.randn(1000,77,400).to(device)

cnn_120 = CNN(77,120,2,).to(device)
cnn_100 = CNN(77,100,2,).to(device)
cnn_50 = CNN(77,50,2,).to(device)

timeit.timeit("test_run(x,cnn_120)",globals=globals(),number=500)
# Output: 18.364156678318977
timeit.timeit("test_run(x,cnn_100)",globals=globals(),number=500)
# Output: 3.4758292948827147
timeit.timeit("test_run(x,cnn_50)",globals=globals(),number=500)
# Output: 23.5902945268899

My assumption is that cnn_100 is so much faster, because it uses the nondeterministic algorithm. The documentation of Conv1d says the following: In some circumstances when using the CUDA backend with CuDNN, this operator may select a nondeterministic algorithm to increase performance.
My questions is now: What requirements has my network to fulfill, so that the nondeterministic algorithm is used?