I would recommend you don’t start from vanilla PyTorch as there are multiple decent libraries built on top of it that simplify building models.
You could try lightning bolts 1 for baseline model architecture + training procedure (you just need to replace the provided dataset with your own. For IoU implementation you can use torchmetrics 4.