Looks like I was able to figure it out. I made the loss very small, 0.00001, and it began converging. I also made sure to set the bias=False in the shortcut convolution layers, although it may have worked even without that. Thank you for your help Peter.