[Fix common tests on GPU] send model, ids to torch_device
~@patrickvonplaten 
 any idea why tf tests dont run? 
https://app.circleci.com/pipelines/github/huggingface/transformers/5734/workflows/7986a344-3a5f-4f4f-a5e7-f8fd82ef7a2d/jobs/33381/steps
seems like test_modeling_common.py only used by torch.~

resolved: had a `PretrainedConfig` type hint 😲 <|||||>Great, thanks for fixing this :-) 