One very easy way to measure time in your program is the snakeviz 25 package. I don’t think it can iron out all questions since it doesn’t really understand GPU stuff but is quick & easy to start with. Nvidia has  GPU-profiler tool but I haven’t got it to work very well
This step initializes your dataloader but doesn’t actually read your data. It shouldn’t take 23 seconds though. Mine takes like a second but idk. 
dataloader = torch.utils.data.DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=NUM_PREPROCESS_WORKERS, drop_last=False, collate_fn=filtered_collate_fn) 


Here you are reading your data and putting it through you model. The time spend is correlated to how much data you have.
for (cids, img_batch, heatmap_batch, tag_batch) in dataloader:
       output_probs = model.extract_mask(img_batch, heatmap_batch)
       q.put((cids, output_probs, tag_batch))

The GPU utilization is dependant on your model & input size. Could you try number_of_workers = 8 with batch_size=16, and then batch_size=32 and report the differences?