A robots.txt file restricts access to your site by search engine
robots that crawl the web. These bots are automated, and before
they access pages of a site, they check to see if a robots.txt
file exists that prevents them from accessing certain pages.
(All respectable robots will respect the directives in a
robots.txt file, although some may interpret them differently.
However, a robots.txt is not enforceable, and some spammers and
other troublemakers may ignore it. For this reason, we recommend
password protecting confidential information.) To see which URLs
Google has been blocked from crawling, visit the Blocked URLs
page of the Health section of Webmaster Tools. You need a
robots.txt file only if your site includes content that you
don't want search engines to index. If you want search engines
to index everything in your site, you don't need a robots.txt
file (not even an empty one). While Google won't crawl or index
the content of pages blocked by robots.txt, we may still index
the URLs if we find them