I get that. I am just speculating if the following calls will help. I am not entirely sure. I’d just try those.
BertModel = BertModel.to(device)
or
self.bert = BertModel.to(device).from_pretrained(         options.model_name_or_path,         from_tf=bool(".ckpt" in options.model_name_or_path),         config=config,         cache_dir=options.cache_dir if options.cache_dir else None,         )