ValueError: could not broadcast input array from shape (2816,384) into shape (2698,384)
the command to reproduce:
cd huggingface-transformers/examples/pytorch/question-answering
python -m torch.distributed.launch --nproc_per_node=8 ./run_qa.py \
		  --model_name_or_path roberta-large \
		  --dataset_name squad  \
		  --do_train --do_eval \
		  --per_device_train_batch_size 16 \
		  --per_device_eval_batch_size 256 \
		  --learning_rate 3e-5 \
		  --num_train_epochs 2 \
		  --max_seq_length 384 \
		  --doc_stride 128 \
		  --output_dir test_result2/$trials --overwrite_output_dir \
		  --logging_dir test_result2/$trials/tensorboard --logging_first_step --logging_steps 50 \
                  --fp16



i tried add "--max_eval_samples 10240", this will fix the error, while the AUC result is quite low(exact_match  = 4.9414, f1 = 8.9784). and when i ran with 1gpu, the above command can succeed(exact_match  = 88.5336, f1 = 94.3266)


the full error is "File "./transformers/src/transformers/trainer_pt_utils.py", line 410, in _nested_set_tensors
    i * slice_len : (i + 1) * slice_len
    i * slice_len : (i + 1) * slice_len
ValueError: could not broadcast input array from shape (2816,384) into shape (2698,384)"
@sgugger i saw you have a PR to fix similar error, could you help to take a look?

<|||||>You should upgrade your version of Transformers: this code is not used anymore inside the `Trainer`.<|||||>i am installing transformers from source, the issue still threr.

and issue throw from https://github.com/huggingface/transformers/blob/86d5fb0b360e68de46d40265e7c707fe68c8015b/src/transformers/trainer_pt_utils.py#L411, looks like the code is still used in master?

<|||||>Indeed, the subclass of the `Trainer` for QA still uses the old code! The PR linked above should fix this.
Thanks for flagging!