Indeed, fighting a good 50 men at once would be rather troublesome, as you wait for the perfect moment to strike. Which does not take long, with Vena wandering off back into your room to look for loot. What bandits are left are busy getting drunk and looking bored. Eventually they start to huddle around and start gambling as well, not even bothering with looking around and actually scouting.