seeing him with lady vernon , however , had put a damper on things , and she now refused to get too close , most likely because she wanted to protect herself from the pain of losing him to another woman ... or other women .