Your approach sounds fine.
You could load a single image, label and the corresponding bounding box in the __getitem__ of your custom Dataset.
I think the easiest way would be to treat this task as a regression use case, i.e. you would provide the coordinates of your bounding boxes as the labels and use a criterion like nn.MSELoss to train your model. The model’s output would be floating point numbers in the desired interval (e.g. normalized to [-1, 1]).