So that’s totally fair and thank you for clarifying. I didn’t want to assume too much and run with it so I tried to temper any comment. 

Honestly, you’re justified in being annoyed. I get it and so should he. That being said, communicating that expectation would be better served perhaps either calmly in the moment (difficult when you’re pissed) or the next morning when(  or day) when you’ve had a chance to simmer down and gather your thoughts. 

With limited time together, wanting those moments to be just the two of you is warranted. I’ll echo what others have said in that you both need to talk and outline expectations for your moments together. 

Wish you all the best and hope talking it out will lead to some positive progress in resolving this!