As you are applying the same convolution to each feature self.conv1_1_1, you are not doing what you showed in your picture. You have to reshape everything in the batch dimension and then apply the convolution. The way you are doing is kind of siamese network more than " in parallel with shared weights"