Answers.Body: <p>It doesn't meet the typical definition of open source, such as <a href="https://opensource.org/osd" rel="nofollow noreferrer">the definition</a> given by the Open Source Initiative, since the <a href="https://www.llama.com/llama3/license/" rel="nofollow noreferrer">llama3 license</a> includes restrictive clauses such as:</p>
  <blockquote>
  <p>If, on the Meta Llama 3 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-07T05:00:07.920Z
Answers.LastActivityDate: 2024-10-07T05:41:00.537Z
Answers.LastEditDate: 2024-10-07T05:41:00.537Z
Answers.UserDisplayName: Franck Dernoncourt
Answers.Body: <p>The main components that <strong>are not available</strong> are: the <em>large data sets of text</em> used to pretrain and fine tune the llama models.</p>
  <p>The components that <strong>are available</strong> are: The <em>software</em> that is used to train and fine tune models. The result of pretraining (the <em>llama base model</em>), the result of fine tuning (the <em>llama instruct and chat models</em>).</p>
  <p>The hardware for pretraining is also very very very expensive. The hardware for fine tuning is possible to rent.</p>
  <p>So 'open source' is slightly misleading, llama is not fully libre in the GNU sense. It is a 'weights available' machine learning model and the source code for training it, fine tuning it, running inference is open source.</p>
  <p>Note: I should clarify a little bit, The exact source code meta used to pretrain their models on their gigantic hardware infrastructure with tens of thousands of GPU nodes is very specific to their individual setup - and this code is not available to us. But there is other open source code available to pretrain models from scratch, including models with the llama architecture.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-26T08:05:26.980Z
Answers.LastActivityDate: 2024-09-26T08:42:23.467Z
Answers.LastEditDate: 2024-09-26T08:42:23.467Z
Answers.UserDisplayName: Aleph 0
Body: <p>I'm trying to understand whether Llama 3 (or other open source models) is fully open-source. Specifically, I would like to know:</p>
  <ul>
  <li>Is the source code for Llama 3 (including the tokenizer, transformers, and other components) available under an open-source license?</li>
  <li>Does the available source code provide everything necessary to build a large language model (LLM) with custom data using Llama 3's architecture?</li>
  </ul>
  <p>For example, if I wanted to train my own model like Llama 3, would I have access to all the underlying code (for tokenization, model architecture, etc.), or are there any proprietary components involved?</p>
  
Tags: <neural-networks><machine-learning><natural-language-processing><large-language-models><open-source>
Title: Is Llama3 fully open-source, including tokenizer, transformers, and other components needed to build a custom LLM?
UserDisplayName: mlibre
ThreadText: # Is Llama3 fully open-source, including tokenizer, transformers, and other components needed to build a custom LLM?
  
  (Asked by: username_0 on 2024-09-26)
  
  I'm trying to understand whether Llama 3 (or other open source models) is fully open-source. Specifically, I would like to know:
  
  * Is the source code for Llama 3 (including the tokenizer, transformers, and other components) available under an open-source license?
  * Does the available source code provide everything necessary to build a large language model (LLM) with custom data using Llama 3's architecture?
  
  For example, if I wanted to train my own model like Llama 3, would I have access to all the underlying code (for tokenization, model architecture, etc.), or are there any proprietary components involved?
  
  **Tags:** neural-networks, machine-learning, natural-language-processing, large-language-models, open-source
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2024-09-26. Score: 3 | Accepted answer
  
  The main components that **are not available** are: the *large data sets of text* used to pretrain and fine tune the llama models.
  
  The components that **are available** are: The *software* that is used to train and fine tune models. The result of pretraining (the *llama base model*), the result of fine tuning (the *llama instruct and chat models*).
  
  The hardware for pretraining is also very very very expensive. The hardware for fine tuning is possible to rent.
  
  So 'open source' is slightly misleading, llama is not fully libre in the GNU sense. It is a 'weights available' machine learning model and the source code for training it, fine tuning it, running inference is open source.
  
  Note: I should clarify a little bit, The exact source code meta used to pretrain their models on their gigantic hardware infrastructure with tens of thousands of GPU nodes is very specific to their individual setup - and this code is not available to us. But there is other open source code available to pretrain models from scratch, including models with the llama architecture.
  
  
  ---
  
  ### Answer by username_1 on 2024-10-07. Score: 3
  
  It doesn't meet the typical definition of open source, such as [the definition](https://opensource.org/osd) given by the Open Source Initiative, since the [llama3 license](https://www.llama.com/llama3/license/) includes restrictive clauses such as:
  
  > If, on the Meta Llama 3 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.
  
  
  ---
  
  