but his displeasure at her departure was aimed largely at her and not the unknown rogue whod usurped his place , bringing him to the conclusion that he was miffed by his memory loss and the fact vanessa had not filled in the gaps .