Show HN: WarcDB: Web crawl data as SQLite databases
Since this is pretty new, some background:WARC is a file format written and read by a rather small but specialized set of web crawler tools, most notably the internet archive's tooling. For example, its java-based crawler heretrix produces warc files.There are a couple of other very cool tools, such as warcprox, which can create web archives from web browser activity by acting as a proxy, pywb which play back the same to make archived versions browsable, and related libraries [1](shoutout to Noah Levitt, Ilya Kreymer and collaborators for building all this).The file format itself is an iso standard, and it's very simple: Capture all http headers sent and received, and all http bodies sent and received, and do simple de-duplication based on digests of the body.There is a companion format, CDX, which builds indexes from warc files (which in turn are just concatenated records, so rather robust).Although all of this is great, I worry a bit about where we're heading with QUIC / udp-based protocols, websockets and other very involved protocols which ultimately make archival much harder.If there's anything you can do to help these (or other) tools to keep our web's archival records alive and flowing, please do so.[1] https://github.com/internetarchive/warcprox