  Obtaining and annotating data can be expensive and time-consuming, especially
in complex, low-resource domains. We use GPT-4 and ChatGPT to augment small
labeled datasets with synthetic data via simple prompts, in three different
classification tasks with varying complexity. For each task, we randomly select
a base sample of 500 texts to generate 5,000 new synthetic samples. We explore
two augmentation strategies: one that preserves original label distribution and
another that balances the distribution. Using a progressively larger training
sample size, we train and evaluate a 110M parameter multilingual language model
on the real and synthetic data separately. We also test GPT-4 and ChatGPT in a
zero-shot setting on the test sets. We observe that GPT-4 and ChatGPT have
strong zero-shot performance across all tasks. We find that data augmented with
synthetic samples yields a good downstream performance, and particularly aids
in low-resource settings, such as in identifying rare classes. Human-annotated
data exhibits a strong predictive power, overtaking synthetic data in two out
of the three tasks. This finding highlights the need for more complex prompts
for synthetic datasets to consistently surpass human-generated ones.

Computation and Language;Computers and Society;Physics and Society