My goal is for the network to accept an arbitrary number of inputs and to learn how to do inference with information from all of the inputs. So if I apply a 1x5 linear transformation to input [[1], [2], [3]], the output will be 3x5 tensor [[a,b,c,d,e],[f,g,h,i,j],[k,l,m,n,o]]. Then if I aggregate the 3 outputs into [[v,w,x,y,z]], the later 5xM layers will learn to use information from all inputs, while early layer will learn to represent them in a way that allows for a useful aggregate.
Am I misunderstanding something? Can you elaborate so I can understand better.
Thank you