If the model fits on a single GPU, you could use a data parallel approach such as nn.DataParallel.
Model sharding could also be used to increase the batch size, but I think a data parallel approach might be easier to apply, since splitting the model is two parts which have approx. an equal memory usage might be more tricky.
Note that if you are usingDistributedDataParallel during the evaluation, you would have to make sure the complete validation set is used only once. By default the DistributedSampler will add extra samples to make the dataset evenly divisible as seen in this line of code 16.