@sgugger created recently a transformation package 309 for fast.ai which seems to be working on images, bounding boxes, segmentation maps etc.
It seems to be the tool you are looking for. As I’m not that familiar with fast.ai I don’t know how easy it would be to use these transformations in PyTorch directly.
Let me know, if that would work for you!