I finally figured a way around this issue. The problem was that we had a group of worker threads launched in the CPU to collect the detached outputs from the GPU and perform some post-processing before saving them to disk.
Let me explain the complete setup:

Launch a set of worker threads which collect images from an asynchronous message queue and feed them to a torch.multiprocessing queue P.
Launch a process task which dequeues items from P and prepares a dataloader and feeds batches of these items to the GPU for inference.
Append items from each batch into a list <item_id, prob> into a torch.multiprocessing queue Q

Launch of set of postprocessing worker threads which collect the items from the Q, do some manipulation in CPU and write them to disk.

With the postprocessing worker threads, my inference rate was around 25 items per second on V100 (10x longer for every batch), when I removed these worker threads, I found that the inference rate increased to 160 records per second. Please note that I manually did the postprocessing before dequeuing the next dataset of items from P- this takes approx 25% of the total time to process a dataset and is essentially idle time for the GPU but I could not improve beyond that.
Story wasnt over yet  ! After removing the postprocesing workers, I was able to get a 6x improvement in inference time with a high GPU memory and usage as shown below. However, do you notice anything odd in the graph below?
￼
￼￼ 
image.png1440×1228 105 KB

The CPU usage keeps increasing over time and before we know the system is bound to crash, once again for no fault of the GPU. Fortunately, this can be resolved by a small trick, by replacing the prob_predict_gpu from prob_predict_gpu.data to prob_predict_gpu.detach.
.data()  is an old api and should not be used anymore. The detach() method constructs a new view on a tensor that is declared not to need gradients  and therefore the subgraph involving the view would not be tracked (nice example here http://www.bnikolic.co.uk/blog/pytorch-detach.html 3)
def mysterious_function(image_batch):
    image_batch_gpu_var = Variable(torch.squeeze(image_batch), requires_grad=False).type(self.dtype_float).cuda(device=self.gpu_id)
    prob_predict_gpu = self.seg_net(image_batch_gpu_var)['probs'] 
    torch.cuda.empty_cache()
    prob_predict_res = prob_predict_gpu.detach.cpu().numpy() #prob_predict_gpu.data.cpu().numpy()
    return prob_predict_res

The final CPU and GPU utilization looked like this:
￼
image.png1440×1204 140 KB