Question: grepping, awking, sedding, and piping are day-to-day routine of a user of any Unix-like operating system, may it be on the command line or inside a shell script (collectively called *filters* from now on).

At their essence, when working with "standard" Unix CLI programs and shell builtins (collectively called *commands* from now on), filters need a precise expected format for stdin, stdout, and stderr in each filter step in order to work correctly. I call this precise expected format of some command an API of this command in the following.

As someone with web development background, I compare this kind of data collecting and data processing technically with [web scraping](http://en.wikipedia.org/wiki/Web_scraping) - a technique which is very instable whenever there is the slightest change in data presentation.

My question now relates to the stability of Unix command APIs.

1. Do commands in a Unix-like operating systems adhere to a formal standardization with respect to their input and output?
2. Have there been instances in history where updates to some important command caused to break the functionality of some filter that was built using an older version of said command?
3. Have Unix commands matured over time that it is absolutely impossible to change in such a way that some filter could break?
4. In case filters may break from time to time due to changing command APIs, how can I as a developer protect my filters against this problem?

Answer: 
First, very brief answers to your questions:

1. Formal standardization of input/output conventions: **no**
2. Breakage in the past due to changing output: **yes**
3. Absolutely impossible to break future filters: **no**
4. How can I protect myself against changes: **be conservative**

When you say "API", you're using a term that (for good or ill) implies too much formality around filter input/output conventions. Very (and I do mean "very") broadly, the primary conventions for data that is ameable to easy filtering are

* each input line is a complete record
* within each record, fields are separated by a known delimiter character

A classic example would be the format of /etc/passwd. But, these default conventions are probably violated to some degree more often than they're followed to the letter.

* There are lots of filters (often written in awk or perl) that parse multiline input formats.
* There are lots of input patterns (eg, /var/log/messages) where there is no well defined field structure, and more general regular expression-based techniques must be used.

Your fourth question, how to protect yourself against variations in output structure, is really the only one that you can do anything about.

* As [@jw013 said](https://unix.stackexchange.com/a/42062/9537), look at what the posix standards say. Of course, posix doesn't specify all commands you'll want to use as input sources.
* If you want your scripts to be portable, try to avoid the idiosyncracies of the whatever version of the some command you happen to have isntalled. For example, many GNU versions of standard unix commands have non-standard extensions. These may be useful, but you should avoid them if you want maximum portability.
* Try to learn what subsets of commands arguments and output formats tend to be stable across platforms. Unfortunately, this requires access to multiple platforms along with time, because these differences won't be written down anywhere, even informally.

In the end, you can't protect yourself fully from the problems you're worried about, and there's no single place to look to for a "definitive" statement of what a certain command should do. For many shell scripts, especially those written for personal or small-scale use, this simply isn't a problem
There are only de facto IO standards — whitespace and null separated output.

As for compatibility, we usually revert to checking version numbers of individual filters. Not that they change much, but when you want to use a brand new feature and still want the script to run on older versions, you have to "ifdef" it out somehow. There is practically no capability reporting mechanism, save for manually writing test cases.