Indeed. In order to train a supervised detector using neural networks you must have labeled bounding boxes on objects. Most detectors have at least weak supervision (which would be the class instead if the bounding box). So by definition, a pretrained detector was trained to detect something. So, in short - I don’t believe you will have much success taking pretrained models off the shelf (without any additional fine tuning or domain adaption) for use in a completely different domain. This is where the data driven NN and classical approaches diverge… Learn from the data, not from human generated heuristics.
As @Steve_Cruz mentioned above you can lower the confidence threshold of yolo, and in my experience that leads to too many false positives to be useful. But combining with some heuristics and some classical vision algorithms (e.g., tracking if there’s video) can give a reasonable result.