Answers.Body: <p>The statement in which you mentioned that &quot;GPT-3 took 405 V100 years to train&quot; refers to the computational resources utilized in training the GPT-3 model. Specifically measured in terms of the equivalent time it would take if the GPT model is trained on a single Nvidia Tesla V100 GPU. To make it more understandable it does not mean that the training process took 405 years, but it indicates the computational intensity or amount of time it would have taken if it was trained on a single Nvidia Tesla V100 GPU.</p>
  <p>Since GPT-3 is a very large model with 175 billion parameters, it requires very high resources in training. According to <a href="https://medium.com/codex/gpt-4-will-be-500x-smaller-than-people-think-here-is-why-3556816f8ff2#:%7E:text=The%20creation%20of%20GPT%2D3,need%20to%20be%20much(!)" rel="nofollow noreferrer">this article</a> it takes around 1,024 Nvidia V100 GPUs to train the model, and it costs around $4.6M and 34 days to train the GPT-3 model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-12-14T10:26:20.643Z
Answers.LastActivityDate: 2023-12-15T13:11:06.897Z
Answers.LastEditDate: 2023-12-15T13:11:06.897Z
Answers.UserDisplayName: Keval Pandya
Body: <p>As described in this <a href="https://arxiv.org/abs/2204.05149" rel="nofollow noreferrer">article</a>, it was written that GPT-3 took 405 V100 years to train in 2020.
  I'm a bit confused about this definition, does that mean the process was accelerated like using a V100 GPU to train in 405 years?</p>
  
Tags: <training><open-ai><chatgpt><gpu><gpt-3>
Title: What is accelerated years in describing the amount of the training time?
UserDisplayName: Marziye Hasanshahi
ThreadText: # What is accelerated years in describing the amount of the training time?
  
  (Asked by: username_0 on 2023-12-14)
  
  As described in this [article](https://arxiv.org/abs/2204.05149), it was written that GPT-3 took 405 V100 years to train in 2020.
  I'm a bit confused about this definition, does that mean the process was accelerated like using a V100 GPU to train in 405 years?
  
  **Tags:** training, open-ai, chatgpt, gpu, gpt-3
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-12-14. Score: 4 | Accepted answer
  
  The statement in which you mentioned that "GPT-3 took 405 V100 years to train" refers to the computational resources utilized in training the GPT-3 model. Specifically measured in terms of the equivalent time it would take if the GPT model is trained on a single Nvidia Tesla V100 GPU. To make it more understandable it does not mean that the training process took 405 years, but it indicates the computational intensity or amount of time it would have taken if it was trained on a single Nvidia Tesla V100 GPU.
  
  Since GPT-3 is a very large model with 175 billion parameters, it requires very high resources in training. According to [this article](https://medium.com/codex/gpt-4-will-be-500x-smaller-than-people-think-here-is-why-3556816f8ff2#:%7E:text=The%20creation%20of%20GPT%2D3,need%20to%20be%20much(!)) it takes around 1,024 Nvidia V100 GPUs to train the model, and it costs around $4.6M and 34 days to train the GPT-3 model.
  
  
  ---
  
  