It looks like you’re missing padding in your convolution to make sure that the output sequence length is the same as the output sequence length. For that you need a padding equal to half the kernel size.
You will still have a mismatch in size in the channel dimension, from what I remember, highway component are usually used for conv that have the same number of input and output channels to avoid this issue. In your case, I guess you can expand the input to be same size as the output if you really want to do this.