If you have a translation-invariant pattern, you might try clever reshaping and using torch.einsum in your forward: In your example, if you have a b x n shaped input for some n==l*s, you can reshape and then for some weight of shape k x l x s do torch.einsum('bls,kls->bks', inp.reshape(b, l, s), weight).view(b, k*s) to get your output.
For irregular patterns masking or using sparse matrix multiplication (with ad-hoc constructed sparse matrices from pre-defined indices) might be a good option.
Best regards
Thomas