Hello,
I have a question about methodology. I have a good dataset (about 8000 samples) which I can use to train a model on an NLP task. As usual, I would normally divide the dataset into training and validation (90% - 10%), or perhaps even training, validation and test datasets (80% - 10% - 10%), and perform training as usual on the training set, then do a hyper parameter search using the performance on the validation set, and once I’m happy with my model parameters, I would run it once on the test set to get the final performance metrics.
Now, assuming (seems a reasonable assumption) that the model will not get worse if we add more training data to it, I was wondering if there is anything wrong with re-training the model with the full dataset once the steps above (validation and testing) have been completed. Obviously you can’t then re-test the model (because the newly trained model will have already seen all the samples in your dataset), but it seems reasonable to say that the performance metric you got on the test set when you only used 80% of the samples for training will not get any worse if you add the remaining 20% of the samples to the training set, hence you can keep the performance you got as a baseline, and then train the model on the full dataset knowing that the performance, if anything, is only going to get better than that.
Is there anything fundamentally wrong with this approach from a scientific or philosophical perspective? Any reason why it shouldn’t be done?