for use by search engines and other crawlers, there is a structured format, the xml sitemap, which lists the pages in a site, their relative importance, and how often they are updated. this is pointed to from the robots.txt file and is typically called sitemap.xml