Thanks a lot, I absolutely missed the fact that workers don’t contribute to the same batch but rather build their own in parallel.
This does solve my issue as now each worker can simply crop the data with a random crop in its collate function and then call the default collate function.
In fact, this certainly also means that the worker could simply do the cropping in the getitem and update the random crop specified by the dataset, as it’s not shared with the other workers, and thus only have to call the function to update the crop coordinates in the collate function.
In the end this second option is much easier, and without changing any of my existing code, I managed to get it working perfectly by having the worker_init_fn and collate_fn do the crop position update:
from torch.utils.data._utils.collate import default_collate

def worker_init(worker_id):
    # Make sure workers don't start with same crop
    torch.utils.data.get_worker_info().dataset.change_random_crop()

def collate(batch):
    dataset = torch.utils.data.get_worker_info().dataset
    print("Collating batch that was cropped with with coordinates:", dataset.top, dataset.left)

    # Change random crop for next batch
    dataset.change_random_crop()
    return default_collate(batch)

Thanks again, this is the elegant solution I was looking for.
Julien