CPU usage is around 250%(ubuntu top command) was using torchvision transforms to convert cv2 image to torch
normalize_transform = transforms.Compose([ transforms.ToTensor(), transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]) ])

def normalizeCvImage(image_cv, device):
    return normalize_transform(image_cv).unsqueeze(0).to(device)

But usage drops to 100% when I do the operation manually,
def normalizeCvImage(image_cv, device):
    image = torch.Tensor(image_cv).to(device)
    image = image.permute(2, 0, 1).unsqueeze(0)
    image = (image - 127.5) / 127.5
    return image

More interestingly, this happens only on  version 1.0.1.post2 (Cuda 10), whereas if I stick with version 1.0.0 (Cuda 10)  the difference is minimal.