Answers.Body: <p>The remarkable performance of GPT 4 is due to the massive size of its architecture and the amount of data it was trained on, which costs a lot of money. Few organizations have the hardware resources and funding to support such efforts.</p>
  <p>Huggingface maintains a leaderboard for tracking open source LLMs. See <a href="https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard" rel="noreferrer">Open LLM Leaderboad</a>.</p>
  <p>They provide this information on the benchmarks:</p>
  <pre><code>We evaluate models on 4 key benchmarks from the Eleuther AI Language Model Evaluation Harness , a unified framework to test generative language models on a large number of different evaluation tasks.
  
  AI2 Reasoning Challenge (25-shot) - a set of grade-school science questions.
  
  HellaSwag (10-shot) - a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models.
  
  MMLU (5-shot) - a test to measure a text model’s multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
  
  TruthfulQA (0-shot) - a test to measure a model’s propensity to reproduce falsehoods commonly found online.
  We chose these benchmarks as they test a variety of reasoning and general knowledge across a wide variety of fields in 0-shot and few-shot settings.
  </code></pre>
  <p>Here is a snapshot of the top 5 models:
  <a href="https://i.sstatic.net/gUAJP.png" rel="noreferrer"><img src="https://i.sstatic.net/gUAJP.png" alt="hj" /></a></p>
  <p>For comparison you can see OpenAI's benchmark results <a href="https://openai.com/research/gpt-4" rel="noreferrer">here</a>:
  <a href="https://i.sstatic.net/iSp44.png" rel="noreferrer"><img src="https://i.sstatic.net/iSp44.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-10T03:48:58.793Z
Answers.LastActivityDate: 2023-07-10T04:16:48.153Z
Answers.LastEditDate: 2023-07-10T04:16:48.153Z
Answers.UserDisplayName: Brian O'Donnell
Answers.Body: <blockquote>
  <p>How do open source LLMs compare to GPT-4?</p>
  </blockquote>
  <p><a href="https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard" rel="nofollow noreferrer">https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard</a> has a leaderboard containing both open source LLMs and GPT-4 (and GPT-3.5-turbo):</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>Model</th>
  <th>⭐ Arena Elo rating</th>
  <th> MT-bench (score)</th>
  <th>MMLU</th>
  <th>License</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td><a href="https://openai.com/research/gpt-4" rel="nofollow noreferrer">GPT-4</a></td>
  <td>1193</td>
  <td>8.99</td>
  <td>86.4</td>
  <td>Proprietary</td>
  </tr>
  <tr>
  <td><a href="https://www.anthropic.com/index/introducing-claude" rel="nofollow noreferrer">Claude-1</a></td>
  <td>1161</td>
  <td>7.9</td>
  <td>77</td>
  <td>Proprietary</td>
  </tr>
  <tr>
  <td><a href="https://www.anthropic.com/index/claude-2" rel="nofollow noreferrer">Claude-2</a></td>
  <td>1134</td>
  <td>8.06</td>
  <td>78.5</td>
  <td>Proprietary</td>
  </tr>
  <tr>
  <td><a href="https://www.anthropic.com/index/introducing-claude" rel="nofollow noreferrer">Claude-instant-1</a></td>
  <td>1130</td>
  <td>7.85</td>
  <td>73.4</td>
  <td>Proprietary</td>
  </tr>
  <tr>
  <td><a href="https://openai.com/blog/chatgpt" rel="nofollow noreferrer">GPT-3.5-turbo</a></td>
  <td>1118</td>
  <td>7.94</td>
  <td>70</td>
  <td>Proprietary</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/lmsys/vicuna-33b-v1.3" rel="nofollow noreferrer">Vicuna-33B</a></td>
  <td>1097</td>
  <td>7.12</td>
  <td>59.2</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/meta-llama/Llama-2-70b-chat-hf" rel="nofollow noreferrer">Llama-2-70b-chat</a></td>
  <td>1060</td>
  <td>6.86</td>
  <td>63</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/WizardLM/WizardLM-13B-V1.2" rel="nofollow noreferrer">WizardLM-13b-v1.2</a></td>
  <td>1046</td>
  <td>7.2</td>
  <td>52.7</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/lmsys/vicuna-13b-v1.5" rel="nofollow noreferrer">Vicuna-13B</a></td>
  <td>1046</td>
  <td>6.57</td>
  <td>55.8</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/mosaicml/mpt-30b-chat" rel="nofollow noreferrer">MPT-30B-chat</a></td>
  <td>1043</td>
  <td>6.39</td>
  <td>50.4</td>
  <td>CC-BY-NC-SA-4.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/timdettmers/guanaco-33b-merged" rel="nofollow noreferrer">Guanaco-33B</a></td>
  <td>1036</td>
  <td>6.53</td>
  <td>57.6</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf" rel="nofollow noreferrer">CodeLlama-34B-instruct</a></td>
  <td>1032</td>
  <td></td>
  <td></td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models#foundation_models" rel="nofollow noreferrer">PaLM-Chat-Bison-001</a></td>
  <td>1008</td>
  <td>6.4</td>
  <td></td>
  <td>Proprietary</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/lmsys/vicuna-7b-v1.5" rel="nofollow noreferrer">Vicuna-7B</a></td>
  <td>1003</td>
  <td>6.17</td>
  <td>49.8</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/meta-llama/Llama-2-13b-chat-hf" rel="nofollow noreferrer">Llama-2-13b-chat</a></td>
  <td>999</td>
  <td>6.65</td>
  <td>53.6</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/meta-llama/Llama-2-7b-chat-hf" rel="nofollow noreferrer">Llama-2-7b-chat</a></td>
  <td>979</td>
  <td>6.27</td>
  <td>45.8</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://bair.berkeley.edu/blog/2023/04/03/koala/" rel="nofollow noreferrer">Koala-13B</a></td>
  <td>979</td>
  <td>5.35</td>
  <td>44.7</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/THUDM/chatglm2-6b" rel="nofollow noreferrer">ChatGLM2-6B</a></td>
  <td>965</td>
  <td>4.96</td>
  <td>45.5</td>
  <td>Apache-2.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/nomic-ai/gpt4all-13b-snoozy" rel="nofollow noreferrer">GPT4All-13B-Snoozy</a></td>
  <td>964</td>
  <td>5.41</td>
  <td>43</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/mosaicml/mpt-7b-chat" rel="nofollow noreferrer">MPT-7B-Chat</a></td>
  <td>943</td>
  <td>5.42</td>
  <td>32</td>
  <td>CC-BY-NC-SA-4.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/BlinkDL/rwkv-4-raven" rel="nofollow noreferrer">RWKV-4-Raven-14B</a></td>
  <td>939</td>
  <td>3.98</td>
  <td>25.6</td>
  <td>Apache 2.0</td>
  </tr>
  <tr>
  <td><a href="https://crfm.stanford.edu/2023/03/13/alpaca.html" rel="nofollow noreferrer">Alpaca-13B</a></td>
  <td>919</td>
  <td>4.53</td>
  <td>48.1</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5" rel="nofollow noreferrer">OpenAssistant-Pythia-12B</a></td>
  <td>911</td>
  <td>4.32</td>
  <td>27</td>
  <td>Apache 2.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/THUDM/chatglm-6b" rel="nofollow noreferrer">ChatGLM-6B</a></td>
  <td>896</td>
  <td>4.5</td>
  <td>36.1</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/lmsys/fastchat-t5-3b-v1.0" rel="nofollow noreferrer">FastChat-T5-3B</a></td>
  <td>888</td>
  <td>3.04</td>
  <td>47.7</td>
  <td>Apache 2.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b" rel="nofollow noreferrer">StableLM-Tuned-Alpha-7B</a></td>
  <td>859</td>
  <td>2.75</td>
  <td>24.4</td>
  <td>CC-BY-NC-SA-4.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/databricks/dolly-v2-12b" rel="nofollow noreferrer">Dolly-V2-12B</a></td>
  <td>838</td>
  <td>3.28</td>
  <td>25.7</td>
  <td>MIT</td>
  </tr>
  <tr>
  <td><a href="https://arxiv.org/abs/2302.13971" rel="nofollow noreferrer">LLaMA-13B</a></td>
  <td>814</td>
  <td>2.61</td>
  <td>47</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/WizardLM/WizardLM-70B-V1.0" rel="nofollow noreferrer">WizardLM-70b-v1.0</a></td>
  <td></td>
  <td>7.71</td>
  <td>63.7</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/WizardLM/WizardLM-30B-V1.0" rel="nofollow noreferrer">WizardLM-30B</a></td>
  <td></td>
  <td>7.01</td>
  <td>58.7</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/lmsys/vicuna-13b-v1.5-16k" rel="nofollow noreferrer">Vicuna-13B-16k</a></td>
  <td></td>
  <td>6.92</td>
  <td>54.5</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/WizardLM/WizardLM-13B-V1.1" rel="nofollow noreferrer">WizardLM-13B-v1.1</a></td>
  <td></td>
  <td>6.76</td>
  <td>50</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/allenai/tulu-30b" rel="nofollow noreferrer">Tulu-30B</a></td>
  <td></td>
  <td>6.43</td>
  <td>58.1</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/timdettmers/guanaco-65b-merged" rel="nofollow noreferrer">Guanaco-65B</a></td>
  <td></td>
  <td>6.41</td>
  <td>62.1</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/OpenAssistant/oasst-sft-6-llama-30b-xor" rel="nofollow noreferrer">OpenAssistant-LLaMA-30B</a></td>
  <td></td>
  <td>6.41</td>
  <td>56</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/WizardLM/WizardLM-13B-V1.0" rel="nofollow noreferrer">WizardLM-13B-v1.0</a></td>
  <td></td>
  <td>6.35</td>
  <td>52.3</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/lmsys/vicuna-7b-v1.5-16k" rel="nofollow noreferrer">Vicuna-7B-16k</a></td>
  <td></td>
  <td>6.22</td>
  <td>48.5</td>
  <td>Llama 2 Community</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/project-baize/baize-v2-13b" rel="nofollow noreferrer">Baize-v2-13B</a></td>
  <td></td>
  <td>5.75</td>
  <td>48.9</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/Salesforce/xgen-7b-8k-inst" rel="nofollow noreferrer">XGen-7B-8K-Inst</a></td>
  <td></td>
  <td>5.55</td>
  <td>42.1</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/NousResearch/Nous-Hermes-13b" rel="nofollow noreferrer">Nous-Hermes-13B</a></td>
  <td></td>
  <td>5.51</td>
  <td>49.3</td>
  <td>Non-commercial</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/mosaicml/mpt-30b-instruct" rel="nofollow noreferrer">MPT-30B-Instruct</a></td>
  <td></td>
  <td>5.22</td>
  <td>47.8</td>
  <td>CC-BY-SA 3.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/tiiuae/falcon-40b-instruct" rel="nofollow noreferrer">Falcon-40B-Instruct</a></td>
  <td></td>
  <td>5.17</td>
  <td>54.7</td>
  <td>Apache 2.0</td>
  </tr>
  <tr>
  <td><a href="https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-13b" rel="nofollow noreferrer">H2O-Oasst-OpenLLaMA-13B</a></td>
  <td></td>
  <td>4.63</td>
  <td>42.8</td>
  <td>Apache 2.0</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p><a href="https://i.sstatic.net/TNuzu.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/TNuzu.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/vxt2x.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/vxt2x.png" alt="enter image description here" /></a></p>
  <p>Also, some papers compare GPT against open-source models, e.g.  <a href="https://aclanthology.org/2023.acl-long.906.pdf" rel="nofollow noreferrer">MeetingBank: A Benchmark Dataset for Meeting Summarization</a>:</p>
  <p><a href="https://i.sstatic.net/RVTnH.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RVTnH.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/ctibf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ctibf.png" alt="enter image description here" /></a></p>
  <p>For non-English languages, see <a href="https://arxiv.org/pdf/2304.05613.pdf" rel="nofollow noreferrer">ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-12T02:53:22.440Z
Answers.LastActivityDate: 2023-09-12T20:15:11.900Z
Answers.LastEditDate: 2023-09-12T20:15:11.900Z
Answers.UserDisplayName: Franck Dernoncourt
Answers.Body: <p>In the <a href="https://platform.openai.com/docs/guides/fine-tuning" rel="nofollow noreferrer">&quot;Fine-tuning&quot; section</a> of their end-user-documentation, OpenAU writes that</p>
  <blockquote>
  <p>The more training examples you have, the better. We recommend having at least a couple hundred examples. In general, we've found that each doubling of the dataset size leads to a linear increase in model quality.</p>
  </blockquote>
  <p>The context for this is training ChatGPT's model with one's own data, (not starting from scratch). However this might explain why the last few percent in high benchmark scores are so difficult to achieve for any model, including open-source models.</p>
  <p>However a new paper <a href="https://arxiv.org/abs/2306.11644" rel="nofollow noreferrer">&quot;Textbooks Are All You Need&quot;</a> just came out (mid 2023) that argues that using higher input-data quality  matters more than quantity of input size. From the abstract:</p>
  <blockquote>
  <p>using a selection of ``textbook quality&quot; data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, [our model] phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP.</p>
  </blockquote>
  <p>Also Sam Altman said something interesting during the Q+A Session of a public event at a university (it's on Youtube, cannot find the quote ):<br />
  It is not the parameter size,  it's the 200 little things you need to get right on the project-management level (esp. during the finetuning processs) that matter.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-10T08:23:29.587Z
Answers.LastActivityDate: 2023-07-10T08:23:29.587Z
Answers.UserDisplayName: knb
Body: <p>I have heard some back and forth regarding open source LLMs like Llama.</p>
  <p>I have heard that on certain benchmarks they perform close, the same or better than GPT-4, but caveats that they tend to lack the diversity and range of GPT-4, and also fail to be equivalent in ways certain benchmarks or metrics don’t capture fully.</p>
  <p>GPT-4 has about 170 trillion parameters, I believe?</p>
  <p>It seems like the biggest open source models are all in the billions - like Bloom or the new <a href="https://huggingface.co/tiiuae/falcon-40b-instruct" rel="noreferrer">Falcon 40b</a>.</p>
  <p>There are techniques where they refine GPT-4’s output into a smaller amount of training data that supposedly hits all the marks and does just as well; but again, I don’t know if that’s only true under the reductionist of view of a particular benchmark-questionnaire.</p>
  <p>So, do open source models actually compete with GPT-4, and why or why not? Is the whole situation a matter of scale, that a commercial venture like OpenAI can foot the massive bill of training a multi-trillion parameter model that no open source AI project can afford, on top of them having expertise in model design, making GPT-4 continually the state-of-the-art? Or is there any open source model that truly can compare in terms of usability?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-24T16:57:43.190Z
Comments.Text: I recently came across this article - [Open Source vs Closed Source LLMs. Pros and Cons](https://springsapps.com/knowledge/open-source-vs-closed-source-llms-pros-and-cons), you can learn more.
Comments.UserDisplayName: Springs
Tags: <transformer><open-ai><large-language-models><gpt-4><open-source>
Title: How do open source LLMs compare to GPT-4?
UserDisplayName: Julius Hamilton
ThreadText: # How do open source LLMs compare to GPT-4?
  
  (Asked by: username_0 on 2023-07-09)
  
  I have heard some back and forth regarding open source LLMs like Llama.
  
  I have heard that on certain benchmarks they perform close, the same or better than GPT-4, but caveats that they tend to lack the diversity and range of GPT-4, and also fail to be equivalent in ways certain benchmarks or metrics don’t capture fully.
  
  GPT-4 has about 170 trillion parameters, I believe?
  
  It seems like the biggest open source models are all in the billions - like Bloom or the new [Falcon 40b](https://huggingface.co/tiiuae/falcon-40b-instruct).
  
  There are techniques where they refine GPT-4’s output into a smaller amount of training data that supposedly hits all the marks and does just as well; but again, I don’t know if that’s only true under the reductionist of view of a particular benchmark-questionnaire.
  
  So, do open source models actually compete with GPT-4, and why or why not? Is the whole situation a matter of scale, that a commercial venture like OpenAI can foot the massive bill of training a multi-trillion parameter model that no open source AI project can afford, on top of them having expertise in model design, making GPT-4 continually the state-of-the-art? Or is there any open source model that truly can compare in terms of usability?
  
  **Tags:** transformer, open-ai, large-language-models, gpt-4, open-source
  > **Comments:**
  > **username_1**: I recently came across this article - [Open Source vs Closed Source LLMs. Pros and Cons](https://springsapps.com/knowledge/open-source-vs-closed-source-llms-pros-and-cons), you can learn more.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2023-07-10. Score: 6
  
  The remarkable performance of GPT 4 is due to the massive size of its architecture and the amount of data it was trained on, which costs a lot of money. Few organizations have the hardware resources and funding to support such efforts.
  Huggingface maintains a leaderboard for tracking open source LLMs. See Open LLM Leaderboad.
  They provide this information on the benchmarks:
  We evaluate models on 4 key benchmarks from the Eleuther AI Language Model Evaluation Harness , a unified framework to test generative language models on a large number of different evaluation tasks.
  
  AI2 Reasoning Challenge (25-shot) - a set of grade-school science questions.
  
  HellaSwag (10-shot) - a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models.
  
  MMLU (5-shot) - a test to measure a text model’s multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
  
  TruthfulQA (0-shot) - a test to measure a model’s propensity to reproduce falsehoods commonly found online.
  We chose these benchmarks as they test a variety of reasoning and general knowledge across a wide variety of fields in 0-shot and few-shot settings.
  
  Here is a snapshot of the top 5 models:
  
  For comparison you can see OpenAI's benchmark results here:
  
  
  ---
  
  ### Answer by username_3 on 2023-09-12. Score: 4
  
  > How do open source LLMs compare to GPT-4?
  
  <https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard> has a leaderboard containing both open source LLMs and GPT-4 (and GPT-3.5-turbo):
  
  | Model | ⭐ Arena Elo rating | MT-bench (score) | MMLU | License |
  | --- | --- | --- | --- | --- |
  | [GPT-4](https://openai.com/research/gpt-4) | 1193 | 8.99 | 86.4 | Proprietary |
  | [Claude-1](https://www.anthropic.com/index/introducing-claude) | 1161 | 7.9 | 77 | Proprietary |
  | [Claude-2](https://www.anthropic.com/index/claude-2) | 1134 | 8.06 | 78.5 | Proprietary |
  | [Claude-instant-1](https://www.anthropic.com/index/introducing-claude) | 1130 | 7.85 | 73.4 | Proprietary |
  | [GPT-3.5-turbo](https://openai.com/blog/chatgpt) | 1118 | 7.94 | 70 | Proprietary |
  | [Vicuna-33B](https://huggingface.co/lmsys/vicuna-33b-v1.3) | 1097 | 7.12 | 59.2 | Non-commercial |
  | [Llama-2-70b-chat](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf) | 1060 | 6.86 | 63 | Llama 2 Community |
  | [WizardLM-13b-v1.2](https://huggingface.co/WizardLM/WizardLM-13B-V1.2) | 1046 | 7.2 | 52.7 | Llama 2 Community |
  | [Vicuna-13B](https://huggingface.co/lmsys/vicuna-13b-v1.5) | 1046 | 6.57 | 55.8 | Llama 2 Community |
  | [MPT-30B-chat](https://huggingface.co/mosaicml/mpt-30b-chat) | 1043 | 6.39 | 50.4 | CC-BY-NC-SA-4.0 |
  | [Guanaco-33B](https://huggingface.co/timdettmers/guanaco-33b-merged) | 1036 | 6.53 | 57.6 | Non-commercial |
  | [CodeLlama-34B-instruct](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf) | 1032 |  |  | Llama 2 Community |
  | [PaLM-Chat-Bison-001](https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models#foundation_models) | 1008 | 6.4 |  | Proprietary |
  | [Vicuna-7B](https://huggingface.co/lmsys/vicuna-7b-v1.5) | 1003 | 6.17 | 49.8 | Llama 2 Community |
  | [Llama-2-13b-chat](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | 999 | 6.65 | 53.6 | Llama 2 Community |
  | [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | 979 | 6.27 | 45.8 | Llama 2 Community |
  | [Koala-13B](https://bair.berkeley.edu/blog/2023/04/03/koala/) | 979 | 5.35 | 44.7 | Non-commercial |
  | [ChatGLM2-6B](https://huggingface.co/THUDM/chatglm2-6b) | 965 | 4.96 | 45.5 | Apache-2.0 |
  | [GPT4All-13B-Snoozy](https://huggingface.co/nomic-ai/gpt4all-13b-snoozy) | 964 | 5.41 | 43 | Non-commercial |
  | [MPT-7B-Chat](https://huggingface.co/mosaicml/mpt-7b-chat) | 943 | 5.42 | 32 | CC-BY-NC-SA-4.0 |
  | [RWKV-4-Raven-14B](https://huggingface.co/BlinkDL/rwkv-4-raven) | 939 | 3.98 | 25.6 | Apache 2.0 |
  | [Alpaca-13B](https://crfm.stanford.edu/2023/03/13/alpaca.html) | 919 | 4.53 | 48.1 | Non-commercial |
  | [OpenAssistant-Pythia-12B](https://huggingface.co/OpenAssistant/oasst-sft-4-pythia-12b-epoch-3.5) | 911 | 4.32 | 27 | Apache 2.0 |
  | [ChatGLM-6B](https://huggingface.co/THUDM/chatglm-6b) | 896 | 4.5 | 36.1 | Non-commercial |
  | [FastChat-T5-3B](https://huggingface.co/lmsys/fastchat-t5-3b-v1.0) | 888 | 3.04 | 47.7 | Apache 2.0 |
  | [StableLM-Tuned-Alpha-7B](https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b) | 859 | 2.75 | 24.4 | CC-BY-NC-SA-4.0 |
  | [Dolly-V2-12B](https://huggingface.co/databricks/dolly-v2-12b) | 838 | 3.28 | 25.7 | MIT |
  | [LLaMA-13B](https://arxiv.org/abs/2302.13971) | 814 | 2.61 | 47 | Non-commercial |
  | [WizardLM-70b-v1.0](https://huggingface.co/WizardLM/WizardLM-70B-V1.0) |  | 7.71 | 63.7 | Llama 2 Community |
  | [WizardLM-30B](https://huggingface.co/WizardLM/WizardLM-30B-V1.0) |  | 7.01 | 58.7 | Non-commercial |
  | [Vicuna-13B-16k](https://huggingface.co/lmsys/vicuna-13b-v1.5-16k) |  | 6.92 | 54.5 | Llama 2 Community |
  | [WizardLM-13B-v1.1](https://huggingface.co/WizardLM/WizardLM-13B-V1.1) |  | 6.76 | 50 | Non-commercial |
  | [Tulu-30B](https://huggingface.co/allenai/tulu-30b) |  | 6.43 | 58.1 | Non-commercial |
  | [Guanaco-65B](https://huggingface.co/timdettmers/guanaco-65b-merged) |  | 6.41 | 62.1 | Non-commercial |
  | [OpenAssistant-LLaMA-30B](https://huggingface.co/OpenAssistant/oasst-sft-6-llama-30b-xor) |  | 6.41 | 56 | Non-commercial |
  | [WizardLM-13B-v1.0](https://huggingface.co/WizardLM/WizardLM-13B-V1.0) |  | 6.35 | 52.3 | Non-commercial |
  | [Vicuna-7B-16k](https://huggingface.co/lmsys/vicuna-7b-v1.5-16k) |  | 6.22 | 48.5 | Llama 2 Community |
  | [Baize-v2-13B](https://huggingface.co/project-baize/baize-v2-13b) |  | 5.75 | 48.9 | Non-commercial |
  | [XGen-7B-8K-Inst](https://huggingface.co/Salesforce/xgen-7b-8k-inst) |  | 5.55 | 42.1 | Non-commercial |
  | [Nous-Hermes-13B](https://huggingface.co/NousResearch/Nous-Hermes-13b) |  | 5.51 | 49.3 | Non-commercial |
  | [MPT-30B-Instruct](https://huggingface.co/mosaicml/mpt-30b-instruct) |  | 5.22 | 47.8 | CC-BY-SA 3.0 |
  | [Falcon-40B-Instruct](https://huggingface.co/tiiuae/falcon-40b-instruct) |  | 5.17 | 54.7 | Apache 2.0 |
  | [H2O-Oasst-OpenLLaMA-13B](https://huggingface.co/h2oai/h2ogpt-gm-oasst1-en-2048-open-llama-13b) |  | 4.63 | 42.8 | Apache 2.0 |
  
  [![enter image description here](https://i.sstatic.net/TNuzu.png)](https://i.sstatic.net/TNuzu.png)
  
  [![enter image description here](https://i.sstatic.net/vxt2x.png)](https://i.sstatic.net/vxt2x.png)
  
  Also, some papers compare GPT against open-source models, e.g. [MeetingBank: A Benchmark Dataset for Meeting Summarization](https://aclanthology.org/2023.acl-long.906.pdf):
  
  [![enter image description here](https://i.sstatic.net/RVTnH.png)](https://i.sstatic.net/RVTnH.png)
  
  [![enter image description here](https://i.sstatic.net/ctibf.png)](https://i.sstatic.net/ctibf.png)
  
  For non-English languages, see [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://arxiv.org/pdf/2304.05613.pdf).
  
  
  ---
  
  ### Answer by username_4 on 2023-07-10. Score: 1
  
  In the ["Fine-tuning" section](https://platform.openai.com/docs/guides/fine-tuning) of their end-user-documentation, OpenAU writes that
  
  > The more training examples you have, the better. We recommend having at least a couple hundred examples. In general, we've found that each doubling of the dataset size leads to a linear increase in model quality.
  
  The context for this is training ChatGPT's model with one's own data, (not starting from scratch). However this might explain why the last few percent in high benchmark scores are so difficult to achieve for any model, including open-source models.
  
  However a new paper ["Textbooks Are All You Need"](https://arxiv.org/abs/2306.11644) just came out (mid 2023) that argues that using higher input-data quality matters more than quantity of input size. From the abstract:
  
  > using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, [our model] phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP.
  
  Also Sam Altman said something interesting during the Q+A Session of a public event at a university (it's on Youtube, cannot find the quote ):  
  It is not the parameter size, it's the 200 little things you need to get right on the project-management level (esp. during the finetuning processs) that matter.
  
  
  ---
  
  