What is the correct method for training NLP models with augmented data? I have a very small dataset (~50 rows) for a text classification problem. I found some open source data that's similar to the problem I'm trying to solve.
Should I...

*

*Train the (BERT) model on the open data, then fine-tune again on the original (small) dataset.

*Combine the two datasets and fine tune the entire thing (also on BERT).

Also, what should be my validation/hold-out set be in either case?

With only 50 samples, I think there is very little one can do. I would suggest training BERT on the "open data" and treating the 50 samples as our hold-out set. Using 50 samples to fine-tuning millions of model parameters is unrealistic. Maybe and depending on the exact task and training data, augmentation could work but only if the samples are quite long and we could split them in some way.
I would advise against combining the two datasets together as this precludes us from using the only "real-world" data we have for testing; we might want to consider synthesizing some data from our real data to fine-tune BERT further but that is a bit of a slippery slope regarding leakage. Ultimately the whole idea is that BERT is pre-trained on such a large amount of data that it already "knows" a lot of information about languages so fine-tuning it at the "open data" should be adequate. So... Option 3. "Original data for hold-out/testing and open data for training/fine-tuning".
