Hi. Since you are not using the pre-trained model weights as a starting point, that’s a possible reason for this performance. There are 2 options here:

You can use the pre-trained model weights and freeze some backbone layers and do a transfer learning with your images. You can do that without even freezing the layers to try it out.
Second would be to increase your dataset size if you don’t want to use the pre-trained model weights and rather train from scratch.
Also to handle for cases like occlusion, try to add more data that represents those cases or use augmentations like CutMix 1, Mosaic etc. which should probably help.
Hope this helps.