Sounds like he doesn’t show you that he appreciates you, and he doesn’t compliment you and help your self esteem.

Honestly it sounds to me that he is using you. Like he actually told you he thought about fucking  your sister? He has fantasies of cheating and “uses” your body to fill these fantasies, but yet he can’t handle the thought of you being with another man (for swinging). Sounds like he’s keeping you around as a sex doll. He needs to actually care about your feelings especially in this situation because a lot of this stems from your self esteem. He has impacted you self esteem extremely and he doesn’t seem to care, that’s just fucked up. 

Personally I’d leave his sorry ass and find a man that loved every bit of me. You should feel wanted, loved, and above that he should make (or at least try to) you feel sexy. 

On  a side note, if you are truly uncomfortable with you lady bits and feel they aren’t up to par than maybe you should consider vaginal rejuvenation, I have heard fantastic things about it, tightens every thing back up.