Multi-GPU memory usage
Hi - I am just using the Seq2SeqTrainer for a fairly vanilla BART fine-tuning use case.  In a multi-GPU setup, the memory, however, is really high for the first GPU (15GB and lower (as expected) for the other 7 (5GB).  What could be causing this?  Here are my training arguments.

>     training_args = Seq2SeqTrainingArguments(
        run_name=args.experiment,
        output_dir='./results',  # output directory
        overwrite_output_dir=True,
        max_steps=20000,  # total number of optimization steps
        per_device_train_batch_size=8,  # batch size per device during training
        per_device_eval_batch_size=8,  # batch size for evaluation
        eval_accumulation_steps=1,
        warmup_steps=0 if args.debug else 200,  # number of warmup steps for learning rate scheduler
        weight_decay=0,  # strength of weight decay
        logging_dir='./logs',  # directory for storing logs
        logging_steps=10,
        logging_first_step=True,
        no_cuda=args.cpu,
        fp16=True,
        do_train=True,
        do_eval=True,
        do_predict=True,
        learning_rate=5e-5,
        dataloader_num_workers=0 if args.debug else 8,
        label_smoothing_factor=0.1,
        save_strategy='steps',
        save_steps=999999 if args.debug else 2500,
        save_total_limit=3,
        seed=1992,
        evaluation_strategy='steps',
        eval_steps=999999 if args.debug else 2500,
        load_best_model_at_end=True,
        metric_for_best_model='mean_f1',
        greater_is_better=True,
    )
cc @sgugger <|||||>How are you launching your script? If you are using `python` and not the `torch.distributed.launch` module, then this is completely normal as the gradients and optimizer state are only stored on GPU 0, not the other ones.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.