We used Atlas to inform our data cleaning and
curation efforts. We started with a collection of ap-
proximately 1,000,000 points. Several data cura-
tion iterations produced our final GPT4All-J train-
ing set. Among other changes, we removed exact
duplicate prompts and responses characterized by
homogeneous clusters in the Atlas map. We also
removed prompts that were less than 10 characters
such as single words like ‘The‘, ‘And‘, as well as
poorly formatted examples.

Interactively explore the cleaned dataset in At-

• Coding questions with a random sub-sample

of Stackoverflow Questions

• Instruction-tuning with a sub-sample of Big-

science/P3

• Custom-generated creative questions.

We accompany this paper with the 800k point
GPT4All-J dataset that is a superset of the origi-
nal 400k points GPT4All dataset. We dedicated
substantial attention to data preparation and cura-
tion.

• GPT4All-J Curated Training Set Map

2 Model Training
2023_GPT4All-J_Technical_Report_2