Can you try printing out output of some layers of your model and checking everything is working fine.
For the second part, you can try to use ‘fold’ function provided by pytorch to concatenate blocks.