Do you have a way to see if the code is bottlenecked? I ran a small experiment to track time taken by each step when processing approx 36k records with batch size of 16.
This step takes 23.6 seconds.
dataloader = torch.utils.data.DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=NUM_PREPROCESS_WORKERS, drop_last=False, collate_fn=filtered_collate_fn) 

This step takes 145 seconds.
for (cids, img_batch, heatmap_batch, tag_batch) in dataloader:
       output_probs = model.extract_mask(img_batch, heatmap_batch)
       q.put((cids, output_probs, tag_batch))