The only adult males allowed in the Harem were the Black Eunuchs, who were in charge of security and administration.
The Black Eunuchs were responsible for the security and administration of the Harem.