Is it wrong to compare multiple models on the same test set and choose the best model? Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter tuning or else we may end up overfitting the test set and have unrealistic results. So it's clearly wrong if we tested some model then try to change its hyperparameters and train, validate, and test it again on the same test set.
However, what if we trained and validated, for example, 5 different models, and we decided that we won't modify any of them again. Then we tested each of the 5 models on the (same) unseen test set. If we select the model that achieves the best test result, isn't this the same as if we try different hyperparameter combinations and select the one that has the best performance on the test set?
In this sense, is it wrong when research papers propose multiple methods, test them on the same data, and decide that one of them is the best because it has the best performance on the test set? Isn't it supposed that the test set doesn't influence the choice of the best method?
But if this is wrong, how are we supposed to compare methods (from different papers) on the same test set without being biased? I feel there is a contradiction regarding this point.

You are correct that the measurement of generalization error should happen after a final model has been selected, on data that has never been passed to any models yet.
This is well explained in Aurelion Geron's book, Hands-on ML with Scikit-Learn, Keras, and Tensorflow. Here's a link to an accompanying Jupyter notebook.
Excerpt (emphasis mine):

Once you are confident about your final model, measure its performance on the test set to estimate the generalization error. Don't tweak your model after measuring the generalization error: you would just start overfitting the test set.

Here, we can consider swapping Model 1 for Model 2 based on the test-set results as a heavy-handed way of "tweaking" the model.
