Yes, that’s possible.
You could either use the functional API with a kernel specified as a plain tensor (not an nn.Parameter):
kernel = torch.randn(...)
out = F.conv2d(input, kernel, ...)

or set the requires_grad attribute of the weight (and bias) in the conv layer to False.