Common Crawl contains the HTML?<p>I wonder how this is legal and considered acceptable. I wish I knew how even Google and others get away with scraping content, saving it, and utilizing it for profit without sharing any revenue with the original webmasters.<p>I know people can opt out of crawling, for those that actually respect that. But still, am I the only one who feels like this is wrong?<p>I guess I have this view that your domain is yours, and you invite the public in like an open house. It&#x27;s my house, my property, and the door is open, where people can come in and look around at my stuff. But the expectation is that only locals will arrive, in small number, and they&#x27;ll be good guests. If someone is breaking the lock on the bedroom door and going through the private drawers, that&#x27;s wrong. If someone is taking photographs of everything, to then create a virtual tour of my house they charge for, that&#x27;s wrong. The expectation is you&#x27;re being nice by providing free and open access to information you created and own, and people should behave courteous to that.<p>Then if you as the webmaster choose, you can provide an API, or database dumps for people to download, along with the licensing terms. That is when it feels right for people to do things like this with the data, because you intentionally provided it through a non-personal interface.<p>To me the web is still a personal interface. I expect humans to use it, in an ordinary human-like way where it is somewhat ephemeral and courteous. I feel like Google cheated their way to success, and Common Crawl is stealing to rise their position in an unfair similar manner.<p>These all seem like parasites to me. They didn&#x27;t create anything, they just steal it en masse.<p>There&#x27;s so many businesses like that, such as Domain Tools that gets rich by hoarding everyone&#x27;s contact details from WHOIS:
<a href="http:&#x2F;&#x2F;whois.domaintools.com&#x2F;commoncrawl.org" rel="nofollow">http:&#x2F;&#x2F;whois.domaintools.com&#x2F;commoncrawl.org</a><p>They have a screenshot history they won&#x27;t ever delete even if you ask nicely. Here is a picture of Common Crawl from 2011:
<a href="http:&#x2F;&#x2F;thumbnails.domaintools.com&#x2F;domaintools&#x2F;2016-01-08T19:20:04.000Z&#x2F;juwC52sCoU2vfhzE8XBhOgzFINQ=&#x2F;commoncrawl.org&#x2F;fullsize&#x2F;39bc3b10013e1f43ccfec67ed654dbd3&#x2F;1322786356.jpg" rel="nofollow">http:&#x2F;&#x2F;thumbnails.domaintools.com&#x2F;domaintools&#x2F;2016-01-08T19:...</a>