Based on the error message it seems that the spatial size of the intermediate activation changed compared to the residual, so you might either want to use a downsampling step or change the architecture. I would also recommend to take a look at the torchvision implementation in case you get stuck.