Soo here it comes
import torch

tensor_in = torch.rand(1, 1, 6, 6, 6)
kernel = torch.zeros(1, 1, 3, 3, 3)  # 3x3x3 kernel
# kernel shape out_channels,in_channels, t,h,w
# Identity kernel

kernel[:, :, 1, 1, 1] = 1

tensor_out = torch.nn.functional.conv3d(tensor_in, kernel, padding=1)

print((tensor_in == tensor_out).all())

First of all think about how to build the identity kernel, which is the one with all zeros but the middle element. This is a just  sanity check to be sure we are properly initilizing the kernel.
If you want to apply your custom kernel to use some smoothing you will have to think in which axis do you want to apply it.
Then choose a proper kernel size which will be 1 for those dimensions in which you aren’t convolving and N for the chosen one.
Lastly be aware of the proper padding size not to downsample the tensor.
In example let’s apply this kernel [-1,1,-1] to the temporal axis:
import torch

tensor_in = torch.rand(1, 1, 6, 6, 6)
kernel = torch.zeros(1, 1, 3, 1, 1)  # 3x1x1 kernel
# kernel shape out_channels,in_channels, t,h,w


kernel[0, 0, :, 0,0] = torch.FloatTensor([-1, 1, -1])


# Padding 1,0,0 required to keep dimensionality
# padding is (1,0,0) in this case
pad = (torch.tensor(kernel.size()[2:]) - 1) // 2
pad = tuple(pad.tolist())
tensor_out = torch.nn.functional.conv3d(tensor_in, kernel, padding=pad)

You can apply the same logic for any axis.
The last but not the least, PyTorch uses Cross-correlation! https://pytorch.org/docs/stable/nn.html#torch.nn.Conv3d 4
So keep that in mind at the time of making kernels.