Hey,
I took a look at your code but can’t see any obvious issues.
You might consider starting with the known-to-work torchvision reference (https://github.com/pytorch/vision/tree/master/references/detection 28) and adjusting it for your use case. I had good luck with this approach.
If you do have more than one image annotated, try using more than one image and a proper Dataset/DataLoader. You shouldn’t have any problems making the model overfit.