Question: I've come across compression algorithms, I tried creating a simple run-length encoding algorithm but I notice that when I searched other algorithms I was shocked to know that some algorithms can process thousands of megabits per second how is this possible? I tried compressing a `1GB` length text it was taking some time for me.

How do is this achievable? or am I missing something here?

[![compression image](https://i.stack.imgur.com/0JVgK.png)](https://i.stack.imgur.com/0JVgK.png)

I can't code well in C, so I created my simple program in Python can these speeds only be available in C?

Answer: 
The short answer to your question is this:

If your goal is *speed* (as it is in typical applications of data compression), then (i) you need to choose a programming language that allows you to write algorithms close to the hardware, and (ii) you will spend a very large amount of time benchmarking and profiling your software to find where it is slow, and then fix these places by using better algorithms and better ways of implementing these algorithms.

The authors of the compression packages you mention have not chosen to use magic to achieve the speed shown in the table. They have just put a large effort (probably years of work) into making their software fast.
There are actually two questions that can be answered:

#### Why is decompression so much faster than compression?

The key point to understand here is, that compression typically involves searching for matching byte sequences, while the compressed format is basically an extremely dense format for instructions on how to create the original byte stream. I.e. while decompression simply means decoding and following the instructions in the compressed file, one step at a time, compression usually means considering many alternatives and selecting one that leads to the smallest amount of required instructions for reconstruction.

A simple corollary of this is, that decompression throughput is significantly higher for highly compressed data (less instructions to decode) than for virtually incompressible data.

#### Why are the standard tools so much faster on doing compression than a naive implementation?

Now that we know that it is the searching for matching byte sequences that consumes most time in a naive compressor implementation, it is clear how the compressors can be optimized: **They achieve their speed by aggressively *avoiding* searching and considering alternatives**. Key part of this is typically some form of hash table, or other quick lookup data structure that allows finding matching sequences rather quickly. And then those standard tools employ heuristics for looking up good matches rather than just matches, allowing them to cut down on the number of matches they will consider before committing to using one of them. One of these heuristics can be that a match is selected when it is "good enough", whatever that means, and that the search continues for longer when the considered matches do not meet this criterion.

As with decompression, each "instruction" in the compressed format requires the compressor to find a (good) match. As such, compression is also fastest for highly compressible data as fewer matches need to be found. Also quite a visible effect when benchmarking compression algorithms.