In special circumstances this would be normal. Commonly with partners that have polar opposite sex drives, or ones that engage in polygamy. 

You should definitely sit him down and discuss this.

If you're okay with him having a mistress but have certain concerns, you should both be able to set some boundaries that you both can realistically adhere to.