I think for your use case you can just use groups=5:
conv = nn.Conv2d(
    in_channels=100,
    out_channels=5,
    kernel_size=3,
    stride=1,
    padding=1,
    groups=5)
print(conv.weight.shape)
> torch.Size([5, 20, 3, 3])

Each kernel of the 5 filters will just use 20 input channels and create an output.
Leaving groups=1 gives you torch.Size([5, 100, 3, 3]), which means each filter will use all 100 input channels.