Which task are you finetuning on?
For sequence to sequence tasks, like summarization, examples/seq2seq/finetune.py supports multigpu for training only. There is a caveat: you have to run the final eval yourself on one GPU.
For language modeling tasks,  multi-gpu is supported through the Trainer class.