Yes, if you have more linear layers after the initial(‘symmetric’)  linear layer you will lose symmetry. You will likely have to constantly apply the same linear layer to the two features, and only concatenate the outputs at the absolute end.