When the Web came out, the early Web crawlers were called robots and to block a robot from accessing your website, to the present day, there's a file called robot.txt, that allows you to deny any Web crawler to access and retrieve that information from your website.