I am training Faster R CNN on custom coco datasets… via this tutorial – https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html 15
and my loss function is getting down into roughly 1.xx.  I then test with some test data, and see a few boxes drawn that are very very wrong and with low confidence scores < .20 … when usually I want to see confidence in >.90
How should I proceed to debug my training/testing ?
Ideas I have so far :

simplify my problem with a simpler dataset and test again
look into my training loss decay to see if I can do better

My loss rate seems to go very very small eventually… I am training a simple step decay of 15, max epochs 400, 674 images in the training set
lr: 0.000000

look at the random images of the groundtruth somewhere in the training, and verify the downsampling/transforms are not breaking the labels/data
switch around to different networks (currently only using resnet50 + resnet101)
*fiddle with hyper parameters like the NMS thresholds, overlap thresholds… based on what I expect the data labels to be, on inference (there should only over be 1 instance of a given class in the picture… for instance)

I’ve had success training prior custom implementations of Faster R CNN, but very much want to test/prove the upgrade to pytorch 1.5 and torchvision .4
How do you all get unstuck ?  Feels like I first need to verify my code with the baseline… would that be the COCO 2017 dataset ?
Thank you!
Edit : Adding a snapshot of the tensorboard, running resnet50

image1856×809 330 KB