Learn just a little Awk (2010)
shell + shell tools + piping = a league of its own.<p>As a big fan and advocate of a low level language (C) and a high level language (Python), as a combo, being a very powerful paradigm, I still can&#x27;t believe some of the things the, shall we call it, shell piping workflow, let&#x27;s you accomplish in one line and in a matter of minutes.<p>Here&#x27;s what you do.<p>Say you have a text log file. It&#x27;s &quot;semi structured&quot; (like most log files) in that you can extract quite a bit of information using things like grep&#x2F;sed&#x2F;awk, but still not fully parse it (without a very complex parser; we don&#x27;t want to get into that). What&#x27;s the first thing you do with the log file? You view it:<p><pre><code>    view logfile
</code></pre>
(view is just opening the file in vim in read-only mode, if you don&#x27;t have it, you can use &quot;vim -R&quot;). You inspect the file and then quit. Try this instead:<p><pre><code>    cat logfile | view -
</code></pre>
(don&#x27;t forget the hyphen at the end). Again, inspect and quit (e.g., using :q). You might say, what a round-about&#x2F;inefficient way to view a logfile. But. Now you can do something in the middle:<p><pre><code>    cat logfile | do something | view -
</code></pre>
inspect and quit. Or more than one thing<p><pre><code>    cat logfile | do something | do something else | view -
</code></pre>
and you keep adding the piped commands until you&#x27;re satisfied with your output. Once you&#x27;re satisfied, you can dump the output into another text file as a &quot;report&quot; by replacing the last &quot;| view -&quot; with &quot;&gt; reportfile&quot;.<p><pre><code>    cat logfile | many | processing | commands | later &gt; reportfile
</code></pre>
and you can do this multiple times to generate multiple report files of various kinds. And you can concatente some of those report files<p><pre><code>    cat file1 file2
</code></pre>
or put some columns of some of the report side by side<p><pre><code>    paste -sd&#x27; &#x27; &lt;(cat file1 | pick column) &lt;(cat file2 | pick column)
</code></pre>
The possibilities are endless.<p>To give one example from my shell history. Some times I have a dozens of pdf files open in my linux desktop (using evince pdf viewer) and I wanna restart the computer but don&#x27;t want to lose track of which pdf files were open. There may be automated ways of doing this but let&#x27;s say there aren&#x27;t any. I start with ps ax:<p><pre><code>    ps ax | view -
</code></pre>
A long log. I want to pick only lines that have evince:<p><pre><code>    ps ax | grep evince | view -
</code></pre>
Now I included the &#x27;grep evince&#x27; line too, which I don&#x27;t want:<p><pre><code>    ps ax | grep evince | grep -v grep | view -
</code></pre>
Good. But I don&#x27;t care about all the columns of ps log, except for the last one (the one that shows the full path of the file). That&#x27;s column 6:<p><pre><code>    ps ax | grep evince | grep -v grep | awk &#x27;{ print $6 }&#x27; | view -
</code></pre>
Looks good. Generate a report file from this:<p><pre><code>    ps ax | grep evince | grep -v grep | awk &#x27;{ print $6 }&#x27; &gt; openpdfs_YYYYMMDD.txt
</code></pre>
Then I close all my pdfs, and reboot. And I don&#x27;t know grep, awk except for very basic things like what I already did. But I know similar tidbits about many other commands. If I have to perform regex search only I use grep, but for search and replace I use sed. Sometimes cut is more handy for column selection than awk. If I don&#x27;t know the command but I know what I need to do, I just google it, and 99% of the time, I can find a stackoverflow post where a linux based one-liner is mentioned which is pretty much a drop-in for my piping workflow.<p>Finally, if I want to traverse through each line of the entry and process one by one, I use the while loop. For example, if instead of dumping the file paths of the pdf I wanted to format it a little bit using directory name and file name, I&#x27;ll do this:<p><pre><code>    ps ax | grep evince | grep -v grep | akw &#x27;{ print $6 }&#x27; | while read -r dfnam; do echo &#x27;Directory:&#x27; $(dirname ${dfnam}) &#x27;File:&#x27; $(basename ${dfnam}); done | view -
</code></pre>
And again, view or dump.<p>Try this, and you would realize that the possibilities are endless.