Would splitting the Datasetsinto anImageDatasetandCoordDataset` work?
You could then use the nested loop approach and keep the image constant in the inner loop.
Pseudo code for the idea:
image_loader = DataLoader(image_dataset)
coord_loader = DataLoader(coord_dataset)

for image in image_loader:
    # if the coordinates depend on the current batch, initialize the coord_loader here
    image = image.to('cuda')
    for coord in coord_loader:
        optimizer.zero_grad()
        coord = coord.to('cuda')
        output = model(image, coord)
        loss = ...

Let me know, if this approach would work or if I misunderstood the use case.