Had he been shouting, that would have been stated in the text; assumption is a big no-no in games like this.  The only outward sign of anger, other than what he said, was him slamming his mug down.  And there's a world of difference between listening for a whisper and trying to pick out a single voice, which isn't raised, in a room that's packed with laughing, talking, shouting, etc etc people.