You could try to increase num_workers of your dataloader for parallelization of data loading. This will decrease the inference time iff your data loading is the bottleneck.
Have you tried to run the inference on a single GPU (without DataParallel)?
Since DataParallel implies some tensor copies, it could also increase your speed for shallow networks.
Another minor issue: you should call torch.cuda.synchronize() every time before you call time.time(). This blocks your program until all cuda ops are finished. Otherwise you might not get a correct timing, since cuda calls are asynchronous and your script may finish on CPU after queueing some cuda calls before the CUDA operations have been executed.