B Training a State-of-the-art Chatbot Experimental Setup Details
B.1 Datasets
We describe the datasets used for QLORA finetuning experiments outlined in Section 5.
OASST1 The OpenAssistant dataset [31] was collected via crowd-sourcing. It contains 161,443
unique messages distributed across 66,497 conversations and spanning 35 different languages. The
dataset often contains several ranked replies for each given user question. In our experiments, we
only use the top reply at each level in the conversation tree. This limits the dataset to 9,209 examples.
We finetuning our models on the full conversation including the user queries.
HH-RLHF This is a human preference dataset about helpfulness and harmlessness. Each datapoint
consists of two assistant replies to a user question along with a human preference judgment of the
best reply. The dataset contains 160,800 examples. When finetuning on this dataset, we combine
QLORA