I think the archive.org crawler should respect robots.txt <i>as it looked at the time of the crawl</i>.  As a well-behaved robot, archive.org&#x27;s crawler should fetch and respect robots.txt each time it crawls.  However, archive.org should <i>not</i> retroactively delete old content when the <i>current</i> site puts up a robots.txt.<p>(To answer your other question, the robots.txt standard already allows giving different instructions to different crawlers.)