I’m not sure I understand your labelling.

I think you want 4 coordinates (x0, y0, h, w).
Are you saying that you have class as an array dimension or do you have class as a separate tensor for each area?
But then computing the IoU per item should be straightforward.
If you don’t have class as an array dimension, but have that as an extra label, using index_add_ using the class as index should help you consolidate per-item IoU to something that has class as an array dimension.

Best regards
Thomas