he and lydia occasionally , in the presence of the servants , made a conventional remark ; otherwise they never spoke ; and the thought in grandcourts mind all the while was of his own infatuation in having given her those diamonds , which obliged him to incur the nuisance of speaking about them .