I had the same question, and for my use in 1d CNN, I wrote a custom convolution layer that creates multiple weights and uses F.conv1d to calculate. This approach might be faster than saving and loading state_dict if performance is in your concern.