Show HN: WarcDB: Web crawl data as SQLite databases
Since this is pretty new, some background:<p>WARC is a file format written and read by a rather small but specialized set of web crawler tools, most notably the internet archive's tooling. For example, its java-based crawler heretrix produces warc files.<p>There are a couple of other very cool tools, such as warcprox, which can create web archives from web browser activity by acting as a proxy, pywb which play back the same to make archived versions browsable, and related libraries [1](shoutout to Noah Levitt, Ilya Kreymer and collaborators for building all this).<p>The file format itself is an iso standard, and it's very simple: Capture all http headers sent and received, and all http bodies sent and received, and do simple de-duplication based on digests of the body.<p>There is a companion format, CDX, which builds indexes from warc files (which in turn are just concatenated records, so rather robust).<p>Although all of this is great, I worry a bit about where we're heading with QUIC / udp-based protocols, websockets and other very involved protocols which ultimately make archival much harder.<p>If there's anything you can do to help these (or other) tools to keep our web's archival records alive and flowing, please do so.<p>[1] <a href="https://github.com/internetarchive/warcprox" rel="nofollow">https://github.com/internetarchive/warcprox</a>