Solved by ptrblck in post #2 
              
              
                Yes, that’s possible. 
You could either use the functional API with a kernel specified as a plain tensor (not an nn.Parameter): 
kernel = torch.randn(...)
out = F.conv2d(input, kernel, ...)

or set the requires_grad attribute of the weight (and bias) in the conv layer to False.