The Elegance of Deflate
Really enjoyed reading this, I think I'd like to look more closely at an implementation to understand it further.Reading the article, I was reminded of a nagging question I've had in the back of my mind for a bit.The ASCII table was invented in the '70s, when the cost of disk storage was much higher than it is today.  But it's a shared dictionary, a standard that we can all agree upon, something that's already on every computer.The thing I've wondered about is whether there could be any advantage to creating new generations of shared dictionaries that are domain-specific, and much larger.For example, in the specific (and over-simplified) case of transmitting large quantities of English text from a server to a client, you could reduce the amount of data sent over the wire if both parties shared a lookup table that contained the entire English dictionary.  In that case, you wouldn't transmit a book as a collection of characters, but rather as a collection of words.Furthermore, it would seem like you could apply the same traditional compression methods like what's described in the article to furthermore reduce the amount of data being sent.  Rather than identifying repeating patterns of letters, you would identify repeating patterns of words.Of course, the obvious drawback is that an English lookup table is useless for transmitting anything other than English text.  But again, disk storage being as cheap as it is, I wonder if it wouldn't be such a monumental problem to store many domain-specific dictionaries.Of course, you'd always want to keep the ASCII table as a fallback.  Much in the same way that a conversation in sign language (ASL, specifically) is largely composed of hand gestures referring to words or concepts, but the language still includes the complete alphabet as a fallback.The thing I don't understand well enough is whether modern compression already incorporates concepts that are similar enough such that a shared-word dictionary would be useless.  It seems like the "LZXX" style of compression essentially includes a similar but dynamically-generated sort of dictionary at the beginning of the transmission, and subsequently refers to that in order to express the message itself.  Would the gains of having enormous shared dictionaries cancel out the advantage of that approach, only in a much more "wasteful" way?