I think you require a .float() at the end. In other words:
class ToTensorWithoutScaling(object):
    """H x W x C -> C x H x W"""
    def __call__(self, image):
        return torch.ByteTensor(np.array(image)).permute(2, 0, 1).float()

This is backed with the source code.
Also, when calling you don’t need the transforms. there:
transform = transforms.Compose([ToTensorWithoutScaling()])