Thanks a lot for your effort. I really appreciate it. However, I don’t think that would work for my case. Assume that I am ranking / sorting given images (or another type of dataset). In that case, I would have N inputs and N outputs, and I would need to preserve the knowledge from each channel to the end. (Feature extraction step of those images would preferably be the same, especially at the beginning of the network.
Thus, they would benefit from using same kernels both in terms of sample efficiency, and in terms of memory requirements, etc). To sum up, in this example problem, I can’t just sum the input channels, because if I sum them, the model wouldn’t be able to sort those documents at the final layer as it wouldn’t have any extracted features specific to each document (which are given as input streams). Am I wrong?