I have the same problem, and I am converting the image to RGB as you suggest. Although really I want a grayscale image.
t_ = None
    if Hyper.is_grayscale:
        t_ = transforms.Compose([
            transforms.Grayscale(num_output_channels=1),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 5])
            ])
    else:
        t_ = transforms.Compose([
            transforms.ToTensor(),
             transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
            ])
    test_img1 = Image.open("test_examples/fast_train.jpg").convert("RGB")
    test_img1 =  t_(test_img1).unsqueeze(0)