Is there a metric to compare machine learning models which takes into account how many trainable parameters they have? Assume we have two models for the same task.
One with 1K parameters and one with 1M and they achieve accuracy as follows




Model
Params
Accuracy




small
1 000
0.8


big
1 000 000
0.9




I would like to compare them is terms of price/time necessary to achieve given accuracy.
Is there some kind of an established metric to compare two models with regard to their size?
Obviously accuracy divided by number of params would be biased towards small models.

It's hard to see how you can compare two models in terms of CPUs, GPUs, RAM, ... if the only information you have is the number of parameters. You can run the same model on hardware with very different cost and running time (different number and type of instances for example) while the accuracy remains the same.
In practice you would distinguish between optimizing metrics and satisficing metrics. Your budget can determine the satisficing metrics price and time. Among models with acceptable cost and running time, you will pick the best model in terms of the optimising metric.
See Andrew Ng's Machine Learning Yearning book.
Updates
Information critera such as AIC and BIC are defined as likelihood + penalty. Basically, if the model isn't fitted by maximizing a likelihood function, information criteria are not applicable.
Akaike Information Criteria applied on Random Forest

Picking appropriate infrastructure for a given model architecture is not trivial. Training and prediction usually have different constraints and so require different infrastructure. The book Learn Amazon SageMaker has practical advice about optimizing cost and performance. Obviously, it's specific to AWS.
