Different train/test for model selection Imagine that I have a function that is somewhat like the following:
def evaluate_model(model, X , y):
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.33, random_state=random())
    model.fit(X_train, y_train)
    print(accuracy_score(y_test, model.predict(X_test)))

And I have several models that I want to check the predictive capability, so I use that function for those models and check the accuracy for them. Am I introducing some kind of bias by using different train/test sets for each model? Suppose that I don't want to use CV for hyperparameter tuning.

If you are introducing a bias then you have a bigger problem. It means that your decision depends on the exact contents of the train and test sets. It shouldn’t. Your decision should be robust to partitioning off the dataset
