After I concatenate them, they will be passed to a few more layers to yield an output. The two feature are not really symmetric in this case.
x1–a few layers–>f(x1)
x2–a few layers–>f(x2)
(f(x1),f(x2),x3,x4)—subsequent layers—>output
So when I switch x1 and x2, it becomes (f(x2),f(x1),x3,x4) which is different and will yield different final results.