Sorry for reviving this old topic but I think this can be a relatively common use case that can help implementing bilateral filter or even learnable input-conditioned filter.
More specific proposal may be to extend F.conv2d so that weight parameter can be a 6D tensor with dimensions (out_channels, in_channels, kH, kW, iH, iW). So the current conv2d producing for the pixel (i, j)
output[i,j] = sum(crop(input, i, j, kH, kW) * weight)

can be generalised to
output[i,j] = sum(crop(input, i, j, kH, kW) * weight[..., i, j])

It is probably difficult to implement it now in a vectorised way without resorting to writing new CUDA kernels.
Another useful API may be generic filter similar to the one in scipy. It allows to avoid re-implementing boundary checks at least.