File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 280, in _setup_backward_hooks     assert p_tmp.grad_fn is not None
Hi
@stas00 @sgugger 
I am testing last version of finetune_trainer on multiple gpus, with this command

`export BS=4; CUDA_VISIBLE_DEVICES=0,1  USE_TF=0   python -m torch.distributed.launch --nproc_per_node=2 --master_port=9910  finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_train --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --sortish_sampler  --task translation --val_max_target_length 128 --warmup_steps 500 --n_train 500 --sharded_ddp
`

these two packages neeed to be added to requirements.txt
pytorch-lightning=1.1.1
fairscale

but still getting this error 

```
12/19/2020 16:56:16 - INFO - utils -   using task specific params for translation: {}
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 291, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 672, in train
    model = ShardedDDP(model, self.optimizer)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 99, in __init__
    self._setup_backward_hooks()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 280, in _setup_backward_hooks
    assert p_tmp.grad_fn is not None
AssertionError
12/19/2020 16:56:16 - INFO - __main__ -   *** Train ***
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 291, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 672, in train
    model = ShardedDDP(model, self.optimizer)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 99, in __init__
    self._setup_backward_hooks()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 280, in _setup_backward_hooks
    assert p_tmp.grad_fn is not None
AssertionError
Traceback (most recent call last):
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/updated/bin/python', '-u', 'finetune_trainer.py', '--local_rank=1', '--model_name_or_path', 't5-small', '--output_dir', 'output_dir', '--adam_eps', '1e-06', '--data_dir', 'wmt_en_ro', '--do_train', '--freeze_embeds', '--label_smoothing', '0.1', '--learning_rate', '3e-5', '--logging_first_step', '--logging_steps', '1000', '--max_source_length', '128', '--max_target_length', '128', '--num_train_epochs', '1', '--overwrite_output_dir', '--per_device_train_batch_size', '4', '--sortish_sampler', '--task', 'translation', '--val_max_target_length', '128', '--warmup_steps', '500', '--n_train', '500', '--sharded_ddp']' returned non-zero exit status 1.
```



thanks 


requirements erorr 
```
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 282, in main
    data_args=data_args,
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/seq2seq_trainer.py", line 58, in __init__
    super().__init__(*args, **kwargs)
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 299, in __init__
    raise ImportError("Sharded DDP training requires fairscale: `pip install fairscale`.")
ImportError: Sharded DDP training requires fairscale: `pip install fairscale`.
Traceback (most recent call last):
  File "finetune_trainer.py", line 353, in <module>
    main()
  File "finetune_trainer.py", line 282, in main
    data_args=data_args,
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/seq2seq_trainer.py", line 58, in __init__
    super().__init__(*args, **kwargs)
  File "/home/rabeeh/ruse/seq2seq/temp/transformer4.1.1/trainer.py", line 299, in __init__
    raise ImportError("Sharded DDP training requires fairscale: `pip install fairscale`.")
ImportError: Sharded DDP training requires fairscale: `pip install fairscale`.
Traceback (most recent call last):
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/updated/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/updated/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/updated/bin/python', '-u', 'finetune_trainer.py', '--local_rank=1', '--model_name_or_path', 't5-small', '--output_dir', 'output_dir', '--adam_eps', '1e-06', '--data_dir', 'wmt_en_ro', '--do_train', '--freeze_embeds', '--label_smoothing', '0.1', '--learning_rate', '3e-5', '--logging_first_step', '--logging_steps', '1000', '--max_source_length', '128', '--max_target_length', '128', '--num_train_epochs', '1', '--overwrite_output_dir', '--per_device_train_batch_size', '4', '--sortish_sampler', '--task', 'translation', '--val_max_target_length', '128', '--warmup_steps', '500', '--n_train', '500', '--sharded_ddp']' returned non-zero exit status 1.
```

As I tried to explain earlier it's not ready yet for general consumption.

If you want it sooner see: https://github.com/huggingface/transformers/issues/9156#issuecomment-748501582