Changing the random_state for the data split results in a changed AUC for my RandomForestClassifier model. Is this expected? I am looking at the AUCs of three RandomForestClassifier models. Before this, I split my data using test and train using a random_state.
When I change the random_state for the data split, the AUCs changes. Is this supposed to happen or does it mean my model most likely needs parameter tuning?
Here are the AUCs using a random_state of 1 and then a random_state of 2.
Model A. 0.76 -> 0.71
Model B. 0.73 -> 0.69
Model C. 0.57 -> 0.58
Thanks.

Without knowing specifically how random state 1 and 2 differ, what's likely happening is that different objects (records, subjects, animals, cars, etc) are being selected for training the the RF classifier.  Once training is done, the out-of-bag (OOB) objects that are not in each bootstrap (for training each tree), are "dropped" down the trained tree to determine class purity after the objects make there way through the nodes.
If the objects used for training and testing change, the AUC can be drastically different.  In addition, there are a variety of CV methods, which I posted here, that will definitely impact AUC.
