Shannon Entropy Imposes Fundamental Limits on Communication
&gt; Engineers can build probabilistic models for patterns of pixel colors from one frame to the next. The models make it possible to calculate the Shannon entropy [...] That value tells you the limit of “lossless” compression — the absolute most the movie can be compressed before you start to lose information about its contents.<p>While this is generally true in practice, strictly speaking the shortest possible encoding of a sequence of bits could be smaller than this would suggest, since it&#x27;s possible a pattern exists that has escaped notice.<p>For example: the first 10 billion digits of pi pass statistical tests for randomness, but can be generated by a short program which calculates them until the desired length is reached, in effect &quot;compressing&quot; them to the length of the generator program.<p>Because of considerations like this, Algorithmic Information Theory[1] equates the information content of a bit sequence with the length of the shortest program which will generate it - though this has the drawback of being generally uncomputable - an intriguingly different paradigm.<p>1: see <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Algorithmic_information_theory" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Algorithmic_information_theory</a>