Hey! Sorry for the late reply, I did actually return to this problem today with much more experience in pytorch and solved it. I wanted to perform gaussian smoothing on a [BATCH x C x H x W] tensor separately for each channel. I created a [kernel_size x kernel_size] gaussian kernel and used the following to fit it as a weight for a depthwise conv2d:
gaussian_kernel = gaussian_kernel.view(1, 1, kernel_size, kernel_size)
gaussian_kernel = gaussian_kernel.repeat(channels, 1, 1, 1)

self.gaussian_filter = nn.Conv2d(in_channels=channels, out_channels=channels,
                                 kernel_size=kernel_size, groups=channels, bias=False)

self.gaussian_filter.weight.data = gaussian_kernel

I know there are some threads on here where people want to perform gaussian smoothing so I will try to post my entire solution in one of those.