Two options:


Use 16 nn.Conv2d modules and manually index the output of s2 to pass it to the correct c3 layers. Use torch.cat at to combine the outputs and then add the biases.


Zero out the c3.weight tensor at the places which aren’t connected. You’ll have to do this after every SGD step too.