Numbers
Firstly, if the Wild Hunt is such a big deal then I would imagine sheer numbers would be enough to overrun most villages and smaller settlements.
Think about what they're taking
You ever seen someone on Methamphetamines? 6 cops can have serious difficulty restraining one person on a lot of meth. Take the right drugs and you will be unphased by a knife through your gut and strong enough to tip a car over. Maybe your elves are high on a very specific cocktail of drugs?
"Successful" Is a subjective term
What actually defines the hunt as successful? Maybe a lot of drow do die. It could be that surviving your first Wild Hunt is a rite of passage. Maybe half the drow that go need serious magic to be healed when they get back.
Animals
Lots of forms of hunting involve animals doing most of the work. Maybe they bring massive beasts that do a lot of the work for them and they bring back badly injured survivors. Maybe they bring something with a paralysing sting and they go 'round collecting people like they're discarded statues.
Tributes
This being a regular event, maybe the world of men use it as an opportunity. Perhaps the drow think they're conquering settlements when actually the only people around are half-starved prisoners who thought being sent out during the Wild Hunt was a good alternative to a life in chains. 164
(Like joining the Night's Watch in GoT). Maybe the cities send out defenceless slaves to lure the drow away from the city?
Influence them
Take some inspiration from the film "The First Purge". Spoilers ahead:
! The guys running the purge had professionals around to ensure the purge was a success. Sure, a few people wanted to kill and stuff, but most people just partied or stayed at home. Throw in a few drow who are either of the ruling class or hired by them that act drunk and crazy but are actually completely clear headed and have them be guiding the other elves. Some of the drow decide to head toward a major city? They pick a fight with them and run off on the other direction and by the time the drunks give up chasing them they've forgotten where they were going before. Some humans are more dangerous than expected? They kill soldiers from the treeline with arrows until the drunks get past their defences.
If all else fails
Just don't address it. A lot of people will imagine a reason far better than you would have put in. People love to speculate. 164