 Cut him off. Who cares what anyone outside immediate family thinks? He should be the one being shamed for his behavior. At the end of your life, it's you and your husband - not that loser and his family. 

 Your happiness is more important than what someone else thinks about you.

Edit: And if anyone asks, you tell them he's been berating you/harassing you and you will not tolerate that.