Answers.Body: <p>ChatGPT is already a model of Large neural networks based on a distributed computing platform. ChatGPT was built in collaboration with Microsoft. The model is trained using Azure supercomputing infrastructure with Nvidia GPUs. ChatGPT’s hardware comprises over 285,000 CPU cores, 10,000 GPUs, and network connectivity of 400 GBs per second per GPU server. Information taken from <a href="https://www.mlyearning.org/does-chatgpt-use-nvidia/" rel="nofollow noreferrer">here</a> and <a href="https://news.microsoft.com/source/features/ai/how-microsofts-bet-on-azure-unlocked-an-ai-revolution/" rel="nofollow noreferrer">here</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-23T18:27:58.503Z
Answers.LastActivityDate: 2023-03-23T19:11:34.057Z
Answers.LastEditDate: 2023-03-23T19:11:34.057Z
Answers.UserDisplayName: Augusto Vasques
Body: <p>Say that we're on GPT20 - maybe the model that's resulted from training is 10PB large (maybe unlikely but this is an example). Is it possible for a GPT model to be distributed across machines? How does this work if distributed inference is needed?</p>
  
Tags: <gpt>
Title: Is it possible for a GPT model to run in a distributed way?
UserDisplayName: alex
ThreadText: # Is it possible for a GPT model to run in a distributed way?
  
  (Asked by: username_0 on 2023-03-23)
  
  Say that we're on GPT20 - maybe the model that's resulted from training is 10PB large (maybe unlikely but this is an example). Is it possible for a GPT model to be distributed across machines? How does this work if distributed inference is needed?
  
  **Tags:** gpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-23. Score: 2
  
  ChatGPT is already a model of Large neural networks based on a distributed computing platform. ChatGPT was built in collaboration with Microsoft. The model is trained using Azure supercomputing infrastructure with Nvidia GPUs. ChatGPT’s hardware comprises over 285,000 CPU cores, 10,000 GPUs, and network connectivity of 400 GBs per second per GPU server. Information taken from [here](https://www.mlyearning.org/does-chatgpt-use-nvidia/) and [here](https://news.microsoft.com/source/features/ai/how-microsofts-bet-on-azure-unlocked-an-ai-revolution/)
  
  
  ---
  
  