I followed your suggestion and ran the code on CPU.
I got the following error:
…./trainer.py", line 1325, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
 “.../modeling_pegasus.py", line 1260, in forward
    return_dict=return_dict,
…..
".../transformers/src/transformers/models/pegasus/modeling_pegasus.py", line 722, in forward
    embed_pos = self.embed_positions(input_shape)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  "..../transformers/models/pegasus/modeling_pegasus.py", line 138, in forward
    return super().forward(positions)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 126, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

Searching in the forum there was a chance that it was related to max_position_embeddings.
The max_position_embedding for distill-pegasus-xsum-16-8 is 512.
My dataset contains one element greater than this value. I removed it and I was able to successfully complete the fine-tuning.
Thanks very much for your help!