in fact , matthew was so reclusive no one outside of the estate had any idea what he looked like and this led to much speculation and many harsh rumors amidst the elite social circuit , linked to both his physical appearance and his demeanor .