You are telling the internet that you would have gone, did you tell him? As a woman you don't have to raise your partner, but from experience I can tell it's good to have some rock solid boundaries. Living together my partner discusses if someone comes over and lets me know if he goes out and listens if I say it's not convenient. That didn't come naturally though. Not at all.

Social anxiety and introverted behaviour is not his framework. You need to tell him. He doesn't discuss it with you because it's clear your problems bother him. Which is normal. Your job is to talk about it with him so he and you can express.