Print the sizes of x1, … x4. Likely, x4 is smaller than the others in the W,H dimensions, that wouldn’t work.
It’s a good idea to implement models from scratch to learn, but you could peek e.g.  at the inception model in torchvision 18 if you are stuck.
Best regards
Thomas