The thing is that supporting such thing is a very large change as each op needs to be reimplemented for it. In particular, if you don’t want to ever allocate the full Tensor, it can be tricky to use libraries like cudnn that do not support these features without a speed drop.
Do you know how tensorflow handle such cases? What is the performance drop of using such structure?
For the case of conv2d, that will depend on which dimension you concatenate over.
If it’s 0, then you can do a for-loop passing each Tensor one by one and then concatenating the outputs along dimension 0.
If it’s 1, then you will need 2 smaller convs (if you have 2 input Tensors) that take a subset of the input channels and the same number of output channels and sum the output of each conv.
If it’s 2 or 3 then it gets quite tricky to do as the interface between the two is shared and you cannot do two independant conv.