Searchable Database of the 183,000 Pirated Books Meta, et al., Used to Train AI
The real tragedy is that people had to work so hard to figure out exactly what was in the dataset. I just never got around to releasing the metadata, which is here: <a href="https:&#x2F;&#x2F;battle.shawwn.com&#x2F;books3-metadata.jsonl" rel="nofollow noreferrer">https:&#x2F;&#x2F;battle.shawwn.com&#x2F;books3-metadata.jsonl</a><p>Dozens of people have told me I belong in prison for making books3. And as far as I can tell, it’s had zero economic impact on authors.<p>If I do end up in prison, I think I’ll join the ranks of &quot;people we look back on throughout history with morbid curiosity&quot;. There are lots of examples of people being imprisoned for things we now consider harmless. The buggery act was used to purge the Knights Templar.<p>See also <a href="https:&#x2F;&#x2F;annas-archive.org&#x2F;llm" rel="nofollow noreferrer">https:&#x2F;&#x2F;annas-archive.org&#x2F;llm</a> for acquiring ~20M books. But at this point you’d better be willing to risk being despised at best.<p>At this point I don’t know why I even did anything in AI. llama-dl was DMCA’ed, books3 is prison fodder, and the big players do as they please with very little risk. Even if the lawsuits are successful, they’ll just pay out damages from VC money.<p>I love ML for the sake of ML. I wanted to prove that individual researchers can contribute in a way that matters. Hopefully they’ll continue to be able to.<p>By the way, the reporter here, Alex Reisner, is a wonderful fellow, and a fellow hacker. He has a bunch of open source projects.