Answers.Body: <p>Yes. There are at least 2 measures of model complexity studied and used in learning theory: <a href="https://mitpress.ublish.com/ebook/foundations-of-machine-learning--2-preview/7093/29" rel="nofollow noreferrer">VC dimension and Rademacher complexity</a>. If you're new to learning theory, you could take a look at <a href="https://ai.stackexchange.com/a/20358/2444">this answer</a>.</p>
  <p>(Note: your question is not an exact duplicate of <a href="https://ai.stackexchange.com/q/17870/2444">this</a>, but the VC dimension is not specific to neural networks).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-16T02:21:33.290Z
Answers.LastActivityDate: 2023-06-16T02:21:33.290Z
Answers.UserDisplayName: nbro
Answers.Body: <p>There is no &quot;unit&quot; to measure model complexity. However, you can take into account some key factors:</p>
  <ul>
  <li><strong>input dimension</strong></li>
  <li><strong>output dimension</strong></li>
  <li><strong>model length (number of layers)</strong>: Model length describes the number of transformations on the data from input to output. A higher number of layers means more possible transformations and thus the possibility to model more complex dependencies.</li>
  <li><strong>model width (number of neurons per layer)</strong>: Model width describes the dimensionality of the data going through a single layer. A higher number of neurons means higher information retention per layer.</li>
  <li><strong>activation functions</strong>: Without any activation functions, even the largest model can only learn linear dependencies. Activation functions introduce non-linearity into the model and make complex dependency modeling possible.</li>
  </ul>
  <p>Based on those factors, you can get a good estimate of how complex your model is. Note that a more complex model does not equal better performance per-se, they require careful fine-tuning.</p>
  <p>To get a good estimate if your model is large enough to model your data transformations, you can train it on very little data with <code>batchsize=1</code> and no regularisation. If it completely learns the given samples, it is likely to be fitting for your entire dataset aswell.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-16T02:08:21.927Z
Answers.Text: The answer address neural network only, not model in general.
Answers.UserDisplayName: lpounng
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T08:39:46.153Z
Answers.LastActivityDate: 2023-06-15T08:39:46.153Z
Answers.UserDisplayName: Quantum
Body: <p>Is there a measure of model complexity?</p>
  
Tags: <deep-learning><models><training-datasets><sample-complexity><vc-theory>
Title: Is there a measure of model complexity?
UserDisplayName: Justaperson
ThreadText: # Is there a measure of model complexity?
  
  (Asked by: username_0 on 2023-06-14)
  
  Is there a measure of model complexity?
  
  **Tags:** deep-learning, models, training-datasets, sample-complexity, vc-theory
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-06-16. Score: 1 | Accepted answer
  
  Yes. There are at least 2 measures of model complexity studied and used in learning theory: [VC dimension and Rademacher complexity](https://mitpress.ublish.com/ebook/foundations-of-machine-learning--2-preview/7093/29). If you're new to learning theory, you could take a look at [this answer](https://ai.stackexchange.com/a/20358/2444).
  
  (Note: your question is not an exact duplicate of [this](https://ai.stackexchange.com/q/17870/2444), but the VC dimension is not specific to neural networks).
  
  
  ---
  
  ### Answer by username_2 on 2023-06-15. Score: 0
  
  There is no "unit" to measure model complexity. However, you can take into account some key factors:
  
  * **input dimension**
  * **output dimension**
  * **model length (number of layers)**: Model length describes the number of transformations on the data from input to output. A higher number of layers means more possible transformations and thus the possibility to model more complex dependencies.
  * **model width (number of neurons per layer)**: Model width describes the dimensionality of the data going through a single layer. A higher number of neurons means higher information retention per layer.
  * **activation functions**: Without any activation functions, even the largest model can only learn linear dependencies. Activation functions introduce non-linearity into the model and make complex dependency modeling possible.
  
  Based on those factors, you can get a good estimate of how complex your model is. Note that a more complex model does not equal better performance per-se, they require careful fine-tuning.
  
  To get a good estimate if your model is large enough to model your data transformations, you can train it on very little data with `batchsize=1` and no regularisation. If it completely learns the given samples, it is likely to be fitting for your entire dataset aswell.
  
  > **Comments:**
  > **username_3**: The answer address neural network only, not model in general.  
  
  ---
  
  