Hi HF/Sagemaker team! I keep getting the following error when I run the run_summarization.py example. Is there any  specific instances type that should I use and why? Thank you.
Error log end:
***** Running training *****
Num examples = 15549
Num Epochs = 10
Instantaneous batch size per device = 2
Total train batch size (w. parallel, distributed & accumulation) = 32
Gradient Accumulation steps = 1
Total optimization steps = 4860
#015  0%|          | 0/4860 [00:00<?, ?it/s]THCudaCheck FAIL file=…/aten/src/THC/THCGeneral.cpp line=139 error=711 : peer mapping resources exhausted
Traceback (most recent call last):
File “run_summarization.py”, line 606, in 
main()
File “run_summarization.py”, line 530, in main
train_result = trainer.train(resume_from_checkpoint=checkpoint)
File “/opt/conda/lib/python3.6/site-packages/transformers/trainer.py”, line 1272, in train
tr_loss += self.training_step(model, inputs)
File “/opt/conda/lib/python3.6/site-packages/transformers/trainer.py”, line 1732, in training_step
loss = self.compute_loss(model, inputs)
File “/opt/conda/lib/python3.6/site-packages/transformers/trainer.py”, line 1766, in compute_loss
outputs = model(**inputs)
File “/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 756, in _call_impl
result = self.forward(*input, **kwargs)
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 157, in forward
inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py”, line 168, in scatter
return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py”, line 37, in scatter_kwargs
kwargs = scatter(kwargs, target_gpus, dim) if kwargs else []
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py”, line 28, in scatter
res = scatter_map(inputs)
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py”, line 19, in scatter_map
return list(map(type(obj), zip(*map(scatter_map, obj.items()))))
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py”, line 15, in scatter_map
return list(zip(*map(scatter_map, obj)))
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py”, line 13, in scatter_map
return Scatter.apply(target_gpus, None, dim, obj)
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/_functions.py”, line 92, in forward
outputs = comm.scatter(input, target_gpus, chunk_sizes, ctx.dim, streams)
File “/opt/conda/lib/python3.6/site-packages/torch/nn/parallel/comm.py”, line 186, in scatter
return tuple(torch._C._scatter(tensor, devices, chunk_sizes, dim, streams))
RuntimeError: cuda runtime error (711) : peer mapping resources exhausted at …/aten/src/THC/THCGeneral.cpp:139
#015  0%|          | 0/4860 [00:13<?, ?it/s]