It makes as much sense as any character you can gin up that hates mages. If you want to make them from some tribal culture that abhors magic, you also could work in some sort of tribal anti-magic folklore or something.