Faster LZ is not the answer to 150-250 GB video game downloads
Rich is a compression expert who wrote a good post, and there are some good comments in this thread already, but a few key summary points to help people understand the tradeoffs here:<p>* Steam, Itch, etc all compress game files before sending them to you over the network for an install. The amount of stuff you download is usually much less than what gets stored on disk. There are still games with unusually massive updates though, especially ones where the files change enough that delta compression fails. This is difficult to avoid when using some middleware due to the design of its file formats. Compressed archives actually <i>make this worse</i> because you can&#x27;t delta compress big archives as well as individual files.<p>* Smaller on disk doesn&#x27;t mean faster to load anymore. In the era of spinning rust HDDs, smaller usually loaded faster, but there were extremes - some compression formats that produced massive size reductions were also 10x slower to decompress or worse, which could produce slower overall game load times. Now that most targets have SSDs, this means that less compression often produces faster loads. See <a href="https:&#x2F;&#x2F;ph3at.github.io&#x2F;posts&#x2F;Asset-Compression&#x2F;" rel="nofollow">https:&#x2F;&#x2F;ph3at.github.io&#x2F;posts&#x2F;Asset-Compression&#x2F;</a> (from elsewhere in this thread) for one example.<p>* For image and geometry data that is headed for the GPU, it&#x27;s best to pick compression formats that can either be natively consumed by the hardware (block texture formats like BC7, etc) or that can be efficiently decompressed on the GPU once you copy it there, like gdeflate. This means the file on disk will probably be bigger, unless you supercompress it using something like Basis. Basis adds measurable CPU time to loading, though, and as a result I&#x27;m considering doing an uncompressed texture cache in my game to improve load times even though I plan to continue shipping textures as basis KTX2.<p>* For streaming asset loads, the CPU usage to decompress the assets during gameplay could lead to frame drops, inconsistent frame pacing, and pop-in. My understanding is that Titanfall opted for uncompressed audio in order to keep CPU usage low when streaming in all the sound effects and voices that could play during a hectic match - at that point in time CPU usage for audio mixing was a serious concern; modern CPUs more headroom so it&#x27;s less of a problem now. It made sense for them to choose this as a trade-off because high-framerate twitch online shooters prioritize low consistent latency.<p>* In some cases the install you get depends on the hardware you have. It&#x27;s not uncommon for older platforms to see &#x27;high res texture packs&#x27; that could be installed optionally. The Xbox platform has something called &quot;smart delivery&quot; where it will deliver a smaller game bundle depending on whether your particular console needs high-def textures. This takes real engineering work to do, so many games opt not to support it because that time is better spent elsewhere.<p>* Ultimately, unless your game is too big to fit on people&#x27;s hard drives at all, there is usually something more important for you to work on vs implementing some complex compression algorithm for every one of your files at rest. You are probably also not as smart as the compression experts who wrote the compression codecs that Steam&#x2F;Itch&#x2F;Xbox&#x2F;Playstation offer, either for install-time compression (to reduce network bandwidth) or texture compression, etc. So in practice you probably just pick an off-the-shelf compression solution and apply it where it improves load times and you leave everything else alone. It doesn&#x27;t make sense to spend weeks shrinking your on-disk size by 5% when you could spend that time fixing crash bugs, improving accessibility with better control customization, or improving the game&#x27;s framerate instead.<p>* Lots of people decide a game &quot;isn&#x27;t compressed&quot; because you can shove it into an lzma archive and make it smaller. This misunderstands the issue. Many compressed or optimized-for-size formats can still be made smaller by using a really heavyweight compression codec like lzma or brotli, but critically, there are tradeoffs involved - slower compression means slower build times means longer development times, and slower decompression means worse load times. The best compression typically also requires you to bundle all your files together, which loses the ability to load only the data you want when you want it.<p>An anecdote: Back when I was working on the original WebAssembly spec, I spent multiple weeks doing lots of research into custom compression and how we could design the format to make it compress more efficiently. I even had some chats with the Brotli team. In the end, we barely did anything, because even months of hard work barely produced better results than just Brotli-compressing our files at maximum settings. We were able to improve gzipped sizes tremendously by filtering and rearranging the data, but there&#x27;s no point doing it if you can just use the browser&#x27;s built in brotli decoder instead.