Yes - in fact, because so few people use Signal, having two people who know each other who you also know appear in your Signal phone book probably means they are talking to each other.  Which might be a problmeatic metadata leak in some cases.