I think something like that will work. However, trimap_alpha is a tensor. I need to convert it to L before I transform it to a tensor. How can I do that?
I forgot to include my data transforms. sorry about that.
data_transforms = {
    'input_training_lowres': transforms.Compose([
        transforms.CenterCrop(224),
        # transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        # transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'gt_training_lowres': transforms.Compose([
        # transforms.Scale(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        # transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'trimap_training_lowres': transforms.Compose([
        # transforms.Scale(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        # transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}