Solved by ptrblck in post #2 
              
              
                You could use a downsample layer as seen in the original ResNet implementation, which could just use a 1x1 conv with the desired number of output channels to match the activation.