What does Bert.py look like ? It’s possible that based on your embedding layers that you are not registering and encapsulating all the attributes in the module so when you try to move the model to the GPU, not all of the attributes move over.
“ > Pytorch requires your to do  self.module_name = module  for things to work correctly. It’s okay to keep them in a list. Just do something like  setattr(self, 'emb_{}'.format(i), emb)  for each step in that loop.
Because I was managing my embedding layers in a list, whereas PyTorch requires all layers be registered as an attribute on the model object, they were not automatically moved over to GPU memory when  model.cuda()  was called. Tricky!”