Can a model overfit on feature position? I am doing a project about predicting which car will win a race given car characteristics as well as road characteristics. I use features corresponding to those : my model takes car1 features, car2 features and road features (I don't care about the pilots yet) to predict whether car1 will win.
Case 1:
I use use the dataset aforementioned.
There are two example rows of the dataset:




car1_winning
car1_feat1
car1_feat2
car2_feat1
car2_feat2
road_feat1




yes
a
b
c
d
e


no
f
g
h
i
j




After training my model using pycaret,I got some decent peformance (best model achieving 69% accuracy).
My model seems to be slighly overfitting on features positions since results differ from two same cars depending on whether I consider one as car1 and the other as car2 : It does not follow  $p_{winning}(car2) = 1-p_{winning}(car1)$.
Case 2:
Since having car2 characteristics in first position (in features) and car1 in second shouldn't impact performance, I flipped cars features (and changed targets labels accordingly) to augment the dataset.
There is an example of resulting dataset:




car1_winning
car1_feat1
car1_feat2
car2_feat1
car2_feat2
road_feat1




yes
a
b
c
d
e


no
f
g
h
i
j


no
c
d
a
b
e


yes
h
i
f
g
j




However, I notice a harsh significant in performance on all the models. My  best model is now achieving now 64% accuracy less than the previous one.
I couldn't find any terminology to check for this behavior, do you have some references ? What steps can be taken to ensure "inversion invariance" ?

It's not totally clear from your question, but I get the impression you're modelling $p(\text{car1 wins}|\text{car1, car2 racing})$? I'll work with that assumption
One option is to think about modelling differently. For example does the performance of car 1 depend on the other car in the race? Maybe it does, this is for you to decide. If not then you could model finishing time as a function of car and road features (independent of the other car), provided you have that information. You can then predict the winner from this.
If performance does actually depend on which two cars are racing, you want to enforce the constraint that $p(\text{car1 wins}|\text{car1, car2 racing})=1-p(\text{car2 wins}|\text{car1, car2 racing})$
ps. by 'achieving 0.64%' I assume you mean your model gets 64% accuracy?
