This won’t work, since you are trying to apply image transformations on coordinates.
Neither ColorJitter not ToTensor should be used on your target.
If you are using RandomCrop, you have to crop the target coordinates in the same way of course.
The easiest way would probably be to use the fuctional API of torchvision.transforms.