neither of them liked to come right out and say these were the happiest years of all , because both remembered their first partners in marriage with love and affection , but in their hearts , both did consider them the happiest .