It should be socially acceptable for Internet Archive to ignore robots.txt.<p>They have to respect it because we, collectively, say so.  Obeying robots.txt is the minimum acceptable behavior for any robot, short of the Asmiov laws.<p>But archiving is different.  I&#x27;ve been running into &quot;Site was not archived due to robots.txt&quot; more and more frequently.  Often these are articles from ~2011 and earlier which the author no doubt would have wanted to be archived.<p>Trouble is, robots.txt is also the only thing that people really bother to set up.  Maybe there&#x27;s a way right now to indicate &quot;Sure, archive my site please, and ignore my robots.txt.&quot;  But if there is, it&#x27;s not really common knowledge, and it&#x27;s kind of unreasonable to expect every single website on the internet to opt-in to that.<p>On the flipside, it seems entirely reasonable that if someone really wants to opt <i>out</i> of archiving, that they explicitly go and tell Internet Archive.  Circa 2016, Internet Archive is the only archive site that seems likely to persist to 2116.  It&#x27;s a shared time capsule, a ship that we all get a free ticket to board.  If someone wants off, they can say so.<p>But right now, large swaths of the internet simply aren&#x27;t being archived due to rules that don&#x27;t entirely seem to make sense.  There are excellent reasons for robot.txt, but opting out of &quot;Make this content available to my children&#x27;s children&#x27;s children&#x27;s children&quot; seems perhaps beyond the scope of the original spec.<p>Would you feel ok with the Archive ignoring your robots.txt, or would you feel annoyed?  If annoyed, then this is a bad idea and should be rejected.<p>But if nobody really cares, then here&#x27;s a proposal:  Internet Archive stops checking &#x2F;robots.txt, and checks for &#x2F;archive.txt instead.  If archive.txt exists, then it&#x27;s parsed and obeyed as if it were a robots.txt file.<p>That way, every site can easily opt-out.  But everyone opts-in by default.  Sites can also exercise control over which portions they want archived, and how often.