Well then yes I agree with your husband. I'm sure he's jealous and insecure (especially if he's not as large as your ex was). 

When you two are having sex he's reminded of how endowed your ex-bf was.

It's literally a visual reminder.

EDIT: For your other reply... I really think you need to look at this from your husbands perspective. You have a tattoo that is a reminder of your ex; it's to scale of how big he was (and is twice the size of your husband); you still miss your ex; and you're unwilling to even compromise and hope he gets over his insecurities (without really being involved in how to make that happen).