But then it should explicitly fails upon loading if run on incompatible hardware, not happily load then generate "stupid things" (per the OP description).
Just a small typo in Continued Pretraining paragraph : "...performance. On text classification (TC)..." should be Token Classification, as Text Classification, in Huggingface's own Transformers library, is more a synonym of Sentence Classification, which is confusing.