Question: I have a *very* large compressed file I'm processing using `grep`. 

```
zcat blah.gz | grep -e pattern | gzip -c > pattern.gz

```

I'd like to print something to `stderr` for every million or so input lines processed, just to keep track of progress. Any ideas?

Answer: 
Run it through awk:

```
zcat blah.gz | awk 'BEGIN {x = 1}; {if ((x++ % 1000000) == 0) print x " " $0 > "/dev/stderr"; print $0 }' | grep -e pattern | gzip -c > pattern.gz

```
The `mbuffer(1)` tool might be something for you. Quoting some abstract:

> 
> The mbuffer tool is used to buffer data streams and show the I/O rate and
>  summary to the user. It is especially useful for writing backups to
>  fast tape drives or streaming them over the network. If used correctly,
>  it can prevent buffer underruns and speed up the whole backup or
>  transfer process.
> 
> 
>