models, we use the mpt-7b-chat model. For Falcon models, we use the Falcon-40B-Instruct model which
is a chat/instruct model. For Vicuna models, we use vicuna-13b-delta-v1.1 and vicuna-33b-delta-v1.3
models from lmsys. All model weights were obtained from HuggingFace.
Since closed-source models have longer context lengths, we change the context length and generation length
to 2000 tokens for these models. To evaluate with closed source models, we collect another set of generations
with 2000 context and generation length.
While collecting generations, we append a system prompt prior to the prompt for evaluation. The system
prompt for each model is shown in Table 31. Since ChatGPT, PaLM, and Falcon do not provide a system
prompt, we use the same system prompt as Llama 2-Chat model. Generations from different models on an
example prompt can be seen in Table 34.
Llama2