In general, torch.nn.Module.forward() 1 doesn’t care if you glue two things together and provide them as single input to the network, and then split and concatenate things inside as you wish. Here’s a small example 10 (DAWNBench CIFAR-10 leader, see def net() in his dawn.py); or lookup for U-Net architecture for splitting and concatenating in big style.