Question: I've come across compression algorithms, I tried creating a simple run-length encoding algorithm but I notice that when I searched other algorithms I was shocked to know that some algorithms can process thousands of megabits per second how is this possible? I tried compressing a `1GB` length text it was taking some time for me.

How do is this achievable? or am I missing something here?

[![compression image](https://i.stack.imgur.com/0JVgK.png)](https://i.stack.imgur.com/0JVgK.png)

I can't code well in C, so I created my simple program in Python can these speeds only be available in C?

Answer: 
I have written [compression software in Rust](https://docs.rs/flate3/0.1.21/flate3/). The answer is not simple, but compression algorithms are usually designed to be reasonably quick. [RFC 1951](https://www.rfc-editor.org/rfc/rfc1951) has two steps, the first stage is to find patterns that occurred earlier in the input, which can be compressed as a pointer to the earlier pattern. This is generally done using a hash lookup.

The second stage is to use Huffman encoding to efficiently represent the values needed for the encoding, using variable (bit) length codes. Values which occur frequently use shorter codes, less frequent values use longer codes.

My software is probably a bit unusual in that it runs the two stages in parallel, which can be faster if your computer has more than one CPU.

There tend to be trade-offs between how much compression is achieved and how fast the compression runs. Finally, the answer is partly that modern computers are **very** fast.
> 
> I can't code well in C, so I created my simple program in Python can these speeds only be available in C?
> 
> 
> 

Yes.

C and possibly C++ are two of the few that are about as close to hardware as most programmers are able to get. (assembly language is not very cross-platform)
Aggressively processing hundreds, thousands, millions, billions, or more of iterations to convert one long bitstream to a smaller bitstream is right at home there.

These things are also tuned every way to Sunday to figure out WHAT they are trying to compress. Is it text, audio (and what format?), video (again what format? mp4, vp6, something decades older. what wrapper? avi, mkv, ogg), microsoft documents, and then there are files that are all of these bundled into a tar (or similar) archive that the user did you the service of not having gzipped the archive to make it harder for you to "squeeze juice" out of? If you are not using metal, or close to metal, something slow like python will suffer no matter how well you tune it.

On the other hand, Python is not that. It is designed for little, short tasks where it doesn't really matter if they are done inefficiently, because they are (relatively) rare. Where with C or C++, finding some little thing on some webpage (oh, and you have to write how to find the thing... and it might miss) to populate a part of a table (oops, the thing is totally too wide for that column because it wasn't the proper hit), all of these difficulties are handled already... with the deal with the devil being speed. For little one-off tasks, speed isn't critical. Millions or trillions of iterations, it will choke.

Then, even if python could "speed itself up" by summoning the functions of the more hardware-centric protocols/methods, it would probably be doing so inefficiently. It would gain on the sub-task, but lose on calling and returning from the sub-task, with maybe a marginal net gain. Then also the "misdiagnosis" where an audio block is requested to be treated as a video block because python misdiagnosed it.

Sorry if this is a bit long-winded.