Error when running run_squad.py in colab
Hi I used the below code which was given as an example:

!python -m torch.distributed.launch --nproc_per_node=8 ./examples/run_squad.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file SQUAD_DIR/train-v1.1.json \
    --predict_file SQUAD_DIR/dev-v1.1.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir wwm_uncased_finetuned_squad/ \
    --per_gpu_eval_batch_size=3   \
    --per_gpu_train_batch_size=3   \

 and I tested it colab notebook but it throwed some error as following:

Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 439, in main
    torch.distributed.init_process_group(backend='nccl')
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/distributed_c10d.py", line 406, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon)
RuntimeError: Address already in use
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
    main()
  File "./examples/run_squad.py", line 437, in main
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
THCudaCheck FAIL file=/pytorch/torch/csrc/cuda/Module.cpp line=33 error=10 : invalid device ordinal
Traceback (most recent call last):
  File "./examples/run_squad.py", line 527, in <module>
    main()
  File "./examples/run_squad.py", line 437, in main
    torch.cuda.set_device(args.local_rank)
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 265, in set_device
    torch._C._cuda_setDevice(device)
RuntimeError: cuda runtime error (10) : invalid device ordinal at /pytorch/torch/csrc/cuda/Module.cpp:33
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 235, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/torch/distributed/launch.py", line 231, in main
    cmd=process.args)
subprocess.CalledProcessError: Command '['/usr/bin/python3', '-u', './examples/run_squad.py', '--local_rank=0', '--model_type', 'bert', '--model_name_or_path', 'bert-large-uncased-whole-word-masking', '--do_train', '--do_eval', '--do_lower_case', '--train_file', 'SQUAD_DIR/train-v1.1.json', '--predict_file', 'SQUAD_DIR/dev-v1.1.json', '--learning_rate', '3e-5', '--num_train_epochs', '2', '--max_seq_length', '384', '--doc_stride', '128', '--output_dir', 'wwm_uncased_finetuned_squad/', '--per_gpu_eval_batch_size=3', '--per_gpu_train_batch_size=3']' returned non-zero exit status 1.


Previously I used other bert package by huggingface (before pytorch-transformers), It worked fine and was fast when used fp16 argument. But after changing it to Pytorch-transformers this is not working.

Can anyone help me in this regard?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
