The temptation of writing shell scripts, illustrated
Bit of questionable advice: I&#x27;ve found using jq for all the argument mangling&#x2F;postprocessing&#x2F;glue tasks instead of sed, awk etc to be pretty workable lately.<p>The -R switch reads in arbitrary text instead of json and presents it as a sequence of json strings internally.<p>The capture() filter applies regexes to the string and extracts subgroups (which will be represented as json objects).<p>The gsub() filter performs search-and-replace on a string.<p>Finally, the -r switch writes the result back to stdout as a raw value, removing any json quoting and escaping.<p>By default, processing is linewise, but there are options to process the whole input at once instead. There is also supposed to be a way how you can have per-line processing with state tracking across lines (like sed&#x27;s &quot;holding space&quot;), but I haven&#x27;t tried that out yet.<p>I found this approach often more flexible than sed and easier to reason about. Also, I believe it may be faster, as more things are happening in a single process.<p>The one exception: grep on files is still orders of magnitude faster than anything else. I believe there also was an article about this on HN a while ago.<p>So if you have to find some data in a larger collection of files and then have to turn the data into something structured, a good approach is to make a &quot;coarse&quot; query with grep or grep -R first, to narrow down your set of candidates, then postprocess your remaining candidates with jq.<p>Also, apropos single process: Another questionable advice is that for&#x2F;while loops in bash are subshells and you can use them as part of a pipe chain. This often lets you reduce the number of processes you need.<p>E.g. instead of<p><pre><code>  for f in $files; do foo &quot;$f&quot; | sed ...; done
</code></pre>
You can often write:<p><pre><code>  for f in $files; do foo &quot;$f&quot;; done | sed ...;
</code></pre>
Which will only launch a single sed process for all files instead of one per file.<p>But yeah, shell scripts do have many shortcomings and in particular the error handling situation seems extremely hard to fix. So actually sensible advice is still: Use python.