Two options
It's too fast
Phone conversations are a little bit awkward. There is missing nonverbal communication that humans rely on to figure out context and meaning. However, the bot's designers were not really smart enough to handle that nonverbal stuff, so when the robot communicates, it relies strictly on voice recognition. The visual systems are not connected to the communication system in any way. Basically, it is a text chatbot that has text to speech and speech to text converters. It also has a limited ability to pick up on verbal cues, like speed and hesitation.
To compensate for this inability, it's got very fast processors. In person, it seems like it's picking up on your body language and facial expressions, but it's actually just thinking really hard about what you said, faster than any human could.*
As a result, all of this information is still available over the phone, and the robot communicates over the phone just as smoothly as it does in person. When two humans have a phone call, it's clumsy: they talk over each other, they have some trouble with the lag, and they slow down because they can't see each other's faces. 209
So when the robot is on the phone, and it sounds the same as it would in person, it's a red flag.
*This does imply that you could detect a robot by showing paradoxical behaviour. If you sound totally friendly, but give the bot a death stare the whole time, it wouldn't pick up on the hostility and would just interpret your words based on how they sound.
It isn't confident about who it's talking to
The bot makes bayesian inferences about who it's talking to at all times, and it's never 100% certain. When it can't see the person it's talking to, it's less confident about who it is. It (probably) won't use the wrong name or anything, but it might shift to a slightly more generic, "public speaking" voice to cover its bases in case it has you wrong. It also might tell you things that it should know you already know, because it isn't 100% sure who you are, e.g. calling the Serson city police chief and saying
I've planted a bomb at the Serson City Hall, it's going to blow up in 30 minutes unless you clear all of your Serson City police officers off of the bridge across the harbour.
A human would call in this threat differently. They wouldn't say "Serson city hall" because the police chief would know which city hall you meant. They would also say "your officers" instead of "your Serson City police officers", and "the bridge" instead of "the bridge across the harbour", because a human would be 100% confident that it's talking to the chief of police. This AI never reaches 100% confidence, and so it hedges its bets. 64