Yes, such networks should work. Your issue is perhaps due to gradient explosion, or something that may not be directly related to a residual summation. In particular, relu without [batch] normalization can cause this (note that successive block outputs can’t decrease in that case).