It's still implying a trust issue. Why not just ask him why he was behaving differently? What if he'd said no, since phones can be a private thing?

Yea, it's good you weren't snooping without consent, but that doesn't *completely* alleviate valid concerns about feeling the need to snoop.