Since you have a mismatch in the channel dimension, you could use an additional conv layer (with a 1x1 kernel) to either increase the channels from 96 to 160 in x_after_conv or decrease the channels to 96 for res.