The large amount of data produced and transmitted daily around the world has resulted in a considerable amount of redundancy in the traffic. Recent studies confirm that most of this redundancy is present at the packet level. In other words, packets generated by the same or different sources, and destined to the same or different clients, contain significant cross-packet correlation. However, for an IP packet with a length only approximately 1500 bytes, conventional compression techniques have proven inefficient in capturing the redundancy in data, as compression performance primarily depends on sequence length. Stated differently, there is a significant penalty with respect to what is fundamentally achievable when attempting to universally compress a finite-length packet. On the other hand, many data packets share common context, which, ideally, could be exploited for improved compression.
For example, it would be desirable to have a computing system capable of encoding and compressing discrete individual delivering packets more efficiently by utilizing the dependency across multiple packets and side-information provided from the memory. Accordingly, a protocol-independent and content-aware network packet compression scheme for removing data redundancy is desired.
Conventional data compression systems have sought to eliminate the redundancy in network data packets by equipping some nodes in the network with memorization capability in order to perform better packet-level redundancy elimination via deduplication. However, these conventional data compression systems may be considered sub-optimal in that they fail to account for either statistical redundancies within a data packet or significant dependencies existing across packets. Thus, it would be desirable to have a data compression system capable of suppressing these statistical redundancies, e.g., via suitable statistical compression techniques.