Hard mining (whether on or offline) is designed to help prevent class imbalance from hindering the network’s ability to learn. Typically in object detection there are way more background pixels than object class pixels, so that’s why it is applied on background. There’s no reason you couldn’t take the same approach for an object class that was overrepresented though, it’s just not as common