Why aren’t you using pretrained models?
Okay, I&#x27;m dead tired of hearing this argument (I&#x27;ve been getting it from reviewers a lot). There&#x27;s a lot of reasons to not use a pretrained model.<p>The most obvious one is that your dataset doesn&#x27;t share a lot of features with the pretrained dataset. This is pretty common in vision tasks where the pretrained set is usually ImageNet. If there isn&#x27;t significant mutual information in the tasks (e.g. language to vision or vise versa), then you aren&#x27;t going to transfer the knowledge that well and you&#x27;ve just wasted your time.<p>Scientific datasets often have this issue as usually you can&#x27;t collect much data and even though there might be shared knowledge between the pre-trained it will give you a bias that you don&#x27;t want or can&#x27;t use. It may even prevent it from generalizing. Training from scratch can smooth this out in some cases. Pretraining means you&#x27;re starting from a different point in the optimization space and it can pigeonhole you towards certain optima.<p>What&#x27;s often being suggested for pre-trained models are HUGE. Sometimes you might as well just write something from scratch (or train a smaller model from scratch) because you just don&#x27;t have the compute. There are many small models that are highly powerful and can be trained from scratch. You can even train transformers on CPUs. Various architectures will help with different tasks, so even just a random pretrained model that does well on a dataset isn&#x27;t going to save you. You may also just be wasting significant&#x2F;costly compute.<p>So when you can, use a pretrained model. But knowledge transfer isn&#x27;t going to always help you. Your millage may vary is all I&#x27;m saying and pretrained models are not the solution to everything.