LsTam91:

done a project in image classification using transf


Here is an idea that occurred to me but was not able to try due to lack of time. I’m not sure if it has been tackled before! My guess is not, but you can do some search to find out.
We report the classification accuracy in any classification problem. This metric, however, might be deceptive due to lots of reasons; data imbalance, the percentage of incorrectly labeled samples (especially in the test set, if any) and the number of classes (the latter would be interesting to tackle).
Now, the simplest problems to consider are CIFAR-10 and CIFAR-100.  I think the best reported performance is by this work 10 and the code is available in PyTorch (EffNet-L2 achieved 99.70% and 96.08 on CIFAR-10 and CIFAR-100, respectively).
The research question: Is EffNet-L2 doing a better job at CIFAR-10 or CIFAR-100?
The problem hence is how to statistically quantify the Model Performance when the number of classes/categories increases? This should be an additional metric that complements the classification accuracy. One dilemma that might affect our judgement is the 10% of samples (which I consider a bit low) used in the testing phase, as it might affect model generalization.