I think you can just apply a convolution using a mask. In the end a convolution is product + sum between the elements  the kernel and the tensor. If you just multiply your tensor by a binary mask (ones for the non-empty values and zeros for empty values) the result should be that.
Besides you have torch.fold and unfold which you can use to emulate an sliding kernel implementing ur own operation.
https://pytorch.org/docs/stable/generated/torch.nn.Fold.html
But basically masking either the input or the output of the convolution should work.
Ooor maybe you can use sparse tensors (Not really aware how deep they have been implemented as i don’t use them).