I’m new to this, so please bare with me. If this is the wrong topic please let me know.
I’ve tried training hard hat detection using both of these datasets:

  

      Roboflow
  

  
    

Hard Hat Universe Object Detection Dataset

  Download 7035 free images labeled with bounding boxes for object detection.


  

  
    
    
  

  


https//universe.roboflow.com/joseph-nelson/hard-hat-workers/3
The problem I’m seeing is that I get A LOT OF detections of hard hats/helmets for normal heads with no helmet. There are a lot more incorrect classifications than correct. I suspect that is due to the fact that all annotations for helmets include the complete head as well.
I guess there are hundres of ways to screw up the training. I’ve been using this template to train, and just replaced the dataset:
colab.research.google.com/drive/1gDZ2xcTOgR39tGGs-EZ6i3RTs16wmzZQ  (link removed as I was not allowed to post multiple links)
Can anyone give any guidance?