Robots.txt meant for search engines don’t work well for web archives
On the linked page, I see comments about ignoring the webmasters' wishes et al.<p>All I can say is f*ck that. It's a free and open internet. If you put content up on a public site, anyone has the right to go and look at it. Stop complaining when someone saves it.<p>And sure some people complain that scrapers slow down their site and that's why they use robots.txt, but really? Really? It's 2017 and your site is affected by that. I think you have bigger things to worry about.