Is there a metric to compare machine learning models which takes into account how many trainable parameters they have? Assume we have two models for the same task.
One with 1K parameters and one with 1M and they achieve accuracy as follows




Model
Params
Accuracy




small
1 000
0.8


big
1 000 000
0.9




I would like to compare them is terms of price/time necessary to achieve given accuracy.
Is there some kind of an established metric to compare two models with regard to their size?
Obviously accuracy divided by number of params would be biased towards small models.

A lot of the classical metrics such as AIC and BIC are made for classical statistical models rather than ML models. In addition classical theory suggest that validation performance is a convex function of model complexity. This is a consequence of the classic variance bias trade-off.
We know this not to be true anymore: deep learning models are so strongly overparameterised that they should perform horribly but they actually perform well. What we observe is a doubly deep descent: with model complexity the validation performance improves, worsens and improves again. This behaviour is not very well understood (to the best of my knowledge). The field of „statistical learning theory“ does tackle this problem, but I am not aware of any metrics that are practically usable as a judgement of model complexity. One such attempt is Rademacher Complexity, which describes how well the model can fit to random data (roughly speaking) thus indicating its potential to overfit. More theory comes from PAC Bayesian theory about generalisation as function of model complexity. However, all of these are very theoretical measures of complexity and I have never seen them applied anywhere.
TL;DR I don’t think there are practical measures of complexity for modern models that are based on statistical theory - equivalent to AIC and BIC.
When it comes to practical cost as a consequence of complexity, there are multiple measures which boil down to: inference latency or model performance as function of training time on a certain device with a certain package. Reading through some recent papers on big, big models will give you an idea of what exactly to calculate.
