According to latest developments, the quality of data is usually as important as quantity. Frequently learning on a small dataset of high quality specialised data can lead to better results than a large dataset of general data.

The training is usually done as a single run, with potential rollbacks to correct loss deterioration. As long as relevant training data is saved in checkpoints, training can be split into multiple runs.

As a large language model I can adapt to new data as long as it is present in my attention window. To gain knowledge on a permanent basis fine-tuning is required.
spam fails_task lang_mismatch pii not_appropriate hate_speech sexual_content quality toxicity humor helpfulness creativity violence