Have you read this?

  
      
      PyTorch Forums – 11 Jan 18
  
  
    

Loss.backward() raises error 'grad can be implicitly created only for scalar... 49

Hi,  loss.backward() do not go through while training and throws an error when on multiple GPUs using torch.nn.DataParallel  grad can be implicitly created only for scalar outputs  But, the same thing trains fine when I give only deviced_ids=[0] to...

  Reading time: 1 mins 🕑
    Likes: 48 ❤

  
  
    
    
  
  


If you haven’t changed the run_clm code, something else must be different. What versions of python, pytorch and huggingface_transformers are you using?
Are you using CPU/GPU/TPU ?  Are you using DataParallel (whatever that is)?