Answers.Body: <p>I hope this helps. Disclaimer: the info is extracted from <a href="https://heartbeat.fritz.ai/computer-vision-at-tesla-cd5e88074376" rel="nofollow noreferrer">Computer Vision at Tesla</a>, though aditional references may be needed....</p>
  <ul>
  <li>Tesla is using 8 cameras per vehicle fused with radars.</li>
  <li>To detect road lanes, vehicles, pedestrians etc., Tesla have to run
  at least 50 neural networks simultaneously.</li>
  <li>The neural networks are trained using PyTorch. Each image (for each
  camera I guess) is of dimension (1280, 960, 3). RGB image it seems.</li>
  <li>Regarding the neural network architecture, &quot;<strong>the backbone is a modified <a href="https://www.kaggle.com/keras/resnet50" rel="nofollow noreferrer">ResNet 50</a>&quot;, that is, a convolutional network.</strong></li>
  <li><a href="https://github.com/matlab-deep-learning/resnet-50" rel="nofollow noreferrer">The network is <strong>50 layers deep</strong>. The network depth is defined as the largest number of sequential convolutional or fully connected layers on a path from the input layer to the output layer. <strong>In total, ResNet-50 has 177 layers</strong></a>. The <a href="https://towardsdatascience.com/understanding-and-coding-a-resnet-in-keras-446d7ff84d33" rel="nofollow noreferrer">ResNet-50 has over 23 million trainable parameters</a></li>
  <li>Every camera is processed through a single neural network. Then everything is combined into a middle neural network.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-25T07:38:49.917Z
Answers.Text: As interesting as this are the hyperparameters and data used to train these networks. Any info on this?
Answers.UserDisplayName: Raul Alvarez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-17T09:22:26.647Z
Answers.LastActivityDate: 2021-05-17T09:41:57.000Z
Answers.LastEditDate: 2021-05-17T09:41:57.000Z
Answers.UserDisplayName: Raul Alvarez
Answers.Body: <p>The size of the model depends on the domain. I am currently working with a model that is used for real time inference on an embedded device. Speed of computation is critical.</p>
  <p>The model size is a 5 layer CNN, about 700k parameters and it's about 12MB in size on disk.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-18T14:22:25.857Z
Answers.LastActivityDate: 2021-05-18T14:22:25.857Z
Answers.UserDisplayName: papiot
Answers.Body: <p>Use the benchmarked algorithms or research papers will be a good start. Addition to that use the open sourced Bert GPT 2 like architectures is a good start.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-13T03:03:09.083Z
Answers.LastActivityDate: 2021-07-13T03:03:09.083Z
Answers.UserDisplayName: slbinilkumar
Answers.Body: <h3>NLP Domain</h3>
  <p>You can easily find such open-source neural networks in NLP applications that have been published by Companies like Google. For example, in <a href="https://github.com/google-research/bert/blob/master/README.md" rel="nofollow noreferrer">BERT models</a>, you can see the BERT-Base has the following specifications:</p>
  <blockquote>
  <p>BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters</p>
  </blockquote>
  <p>You can find more data about other versions of BERT in the same link.</p>
  <p>Another example is GPT models, like <a href="https://lambdalabs.com/blog/demystifying-gpt-3" rel="nofollow noreferrer">GPT-3</a>:</p>
  <blockquote>
  <p>All GPT-3 models use the same attention-based architecture as their GPT-2 predecessor. The smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads. The largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.</p>
  </blockquote>
  <h3>Image Procssing Domain</h3>
  <p>Another useful domain for your expectation is image processing tasks such as image classification. Pretrained models such as <a href="https://www.analyticsvidhya.com/blog/2020/08/top-4-pre-trained-models-for-image-classification-with-python-code/" rel="nofollow noreferrer">VGG, ResNet, and Inception</a>. These are mostly used for image classification tasks in different companies, and you can find their specification many where. For example for VGG-16, we can see the followings:</p>
  <p><a href="https://i.sstatic.net/YekFP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YekFP.png" alt="enter image description here" /></a></p>
  <h3>Speech Processing</h3>
  <p>Another practical domain is Auto-Speech Recognition or ASR in short. One of the renowned models in this context is DeepSpeech(2) by Baido Research center. For example, you can find some info like the number of its parameters and its structure in <a href="https://gitlab.com/Jaco-Assistant/Scribosermo" rel="nofollow noreferrer">this github link</a>.</p>
  <h3>Summing up</h3>
  <p>Note that one regular metric to measure the size of neural networks is &quot;<strong>the number of parameters</strong>&quot; of the network that is required to be learned in the training phase. Hence, you can compare the size of models even between cross domains by knowing their number of parameters (instead of going to more details about the number of hidden layers and their types). Although, sometimes the length (number of layers) and height (number of neurons in each layer) of the network are very important in the matter of performance and capability of the network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-16T16:18:03.703Z
Answers.LastActivityDate: 2021-05-17T22:00:59.893Z
Answers.LastEditDate: 2021-05-17T22:00:59.893Z
Answers.UserDisplayName: OmG
Body: <p>I'm interested in artificial neural networks (ANN) and I wonder how big ANNs in practical use are, for example, Tesla Autopilot, Google Translate, and others.</p>
  <p>The only thing I found about Tesla is <a href="https://www.tesla.com/autopilotAI" rel="nofollow noreferrer">this one</a>:</p>
  <blockquote>
  <p>&quot;A full build of Autopilot neural networks involves 48 networks that
  take 70,000 GPU hours to train. Together, they output 1,000 distinct
  tensors (predictions) at each timestep.&quot;</p>
  </blockquote>
  <p>It seems like most companies don't publish clear information about their ANN sizes. I really can't find anything detailed on this subject.</p>
  <p>Is there any information about the size of big practical/commercial ANNs that include something like the amount of neurons/connections/layers etc.?</p>
  <p>I'm looking for a few <em>examples</em> in this scale with more precise information on the size of the neural networks.</p>
  
Tags: <neural-networks><machine-learning><reference-request><autonomous-vehicles>
Title: What are the typical sizes of practical/commercial artificial neural networks?
UserDisplayName: Mitarano
ThreadText: # What are the typical sizes of practical/commercial artificial neural networks?
  
  (Asked by: username_0 on 2021-05-16)
  
  I'm interested in artificial neural networks (ANN) and I wonder how big ANNs in practical use are, for example, Tesla Autopilot, Google Translate, and others.
  
  The only thing I found about Tesla is [this one](https://www.tesla.com/autopilotAI):
  
  > "A full build of Autopilot neural networks involves 48 networks that
  > take 70,000 GPU hours to train. Together, they output 1,000 distinct
  > tensors (predictions) at each timestep."
  
  It seems like most companies don't publish clear information about their ANN sizes. I really can't find anything detailed on this subject.
  
  Is there any information about the size of big practical/commercial ANNs that include something like the amount of neurons/connections/layers etc.?
  
  I'm looking for a few *examples* in this scale with more precise information on the size of the neural networks.
  
  **Tags:** neural-networks, machine-learning, reference-request, autonomous-vehicles
  
  ---
  
  ## 4 answers
  
  ### Answer by username_4 on 2021-05-16. Score: 3 | Accepted answer
  
  ### NLP Domain
  
  You can easily find such open-source neural networks in NLP applications that have been published by Companies like Google. For example, in [BERT models](https://github.com/google-research/bert/blob/master/README.md), you can see the BERT-Base has the following specifications:
  
  > BERT-Base, Multilingual Cased: 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters
  
  You can find more data about other versions of BERT in the same link.
  
  Another example is GPT models, like [GPT-3](https://lambdalabs.com/blog/demystifying-gpt-3):
  
  > All GPT-3 models use the same attention-based architecture as their GPT-2 predecessor. The smallest GPT-3 model (125M) has 12 attention layers, each with 12x 64-dimension heads. The largest GPT-3 model (175B) uses 96 attention layers, each with 96x 128-dimension heads.
  
  ### Image Procssing Domain
  
  Another useful domain for your expectation is image processing tasks such as image classification. Pretrained models such as [VGG, ResNet, and Inception](https://www.analyticsvidhya.com/blog/2020/08/top-4-pre-trained-models-for-image-classification-with-python-code/). These are mostly used for image classification tasks in different companies, and you can find their specification many where. For example for VGG-16, we can see the followings:
  
  [![enter image description here](https://i.sstatic.net/YekFP.png)](https://i.sstatic.net/YekFP.png)
  
  ### Speech Processing
  
  Another practical domain is Auto-Speech Recognition or ASR in short. One of the renowned models in this context is DeepSpeech(2) by Baido Research center. For example, you can find some info like the number of its parameters and its structure in [this github link](https://gitlab.com/Jaco-Assistant/Scribosermo).
  
  ### Summing up
  
  Note that one regular metric to measure the size of neural networks is "**the number of parameters**" of the network that is required to be learned in the training phase. Hence, you can compare the size of models even between cross domains by knowing their number of parameters (instead of going to more details about the number of hidden layers and their types). Although, sometimes the length (number of layers) and height (number of neurons in each layer) of the network are very important in the matter of performance and capability of the network.
  
  
  ---
  
  ### Answer by username_1 on 2021-05-17. Score: 1
  
  I hope this helps. Disclaimer: the info is extracted from [Computer Vision at Tesla](https://heartbeat.fritz.ai/computer-vision-at-tesla-cd5e88074376), though aditional references may be needed....
  
  * Tesla is using 8 cameras per vehicle fused with radars.
  * To detect road lanes, vehicles, pedestrians etc., Tesla have to run
    at least 50 neural networks simultaneously.
  * The neural networks are trained using PyTorch. Each image (for each
    camera I guess) is of dimension (1280, 960, 3). RGB image it seems.
  * Regarding the neural network architecture, "**the backbone is a modified [ResNet 50](https://www.kaggle.com/keras/resnet50)", that is, a convolutional network.**
  * [The network is **50 layers deep**. The network depth is defined as the largest number of sequential convolutional or fully connected layers on a path from the input layer to the output layer. **In total, ResNet-50 has 177 layers**](https://github.com/matlab-deep-learning/resnet-50). The [ResNet-50 has over 23 million trainable parameters](https://towardsdatascience.com/understanding-and-coding-a-resnet-in-keras-446d7ff84d33)
  * Every camera is processed through a single neural network. Then everything is combined into a middle neural network.
  
  > **Comments:**
  > **username_1**: As interesting as this are the hyperparameters and data used to train these networks. Any info on this?  
  
  ---
  
  ### Answer by username_2 on 2021-05-18. Score: 0
  
  The size of the model depends on the domain. I am currently working with a model that is used for real time inference on an embedded device. Speed of computation is critical.
  
  The model size is a 5 layer CNN, about 700k parameters and it's about 12MB in size on disk.
  
  
  ---
  
  ### Answer by username_3 on 2021-07-13. Score: 0
  
  Use the benchmarked algorithms or research papers will be a good start. Addition to that use the open sourced Bert GPT 2 like architectures is a good start.
  
  
  ---
  
  