text: You could be making yourself too obviously interested for his comfort.  You know, I'm sure, how workplace rumors can crush a persons reputation and perhaps he's trying to avoid anyone witnessing anything that will look bad for the both of you in the long run.