How does the substitution jutsu work? Almost every character in the Naruto series, from minor character to boss level character, uses this jutsu (Kawarimi no Jutsu).

Ever since I started watching the series, I wanted to know how this technique works. It may look like a Ninjutsu 101 technique in the early parts of the series, but in many of the serious fights later in the series, it seems to fool even the opponents who are well-versed in combat. Is this jutsu based on any real life technique?
According to this, it is indeed based on a real-life technique:


  This technique is based on a real-life ninja art, kawarimi. It was an
  ancient ninja art revolving around misdirection and the split-second
  timing of a switch between a 'body' and a location, or between two or
  more bodies: A technique utilized by ninja in ancient Japan, involving
  the ninja and anything that can be mistaken for them, (such as a dummy
  or a mannequin).


As you mention, it is used by all levels of characters, because it is one of the basic techniques being taught in the Academy. 

Now, for the "it seems to fool even the opponents who are well versed in combat" part. This is also similar to the real word. For example, all boxers use pretty much the same punches, yet some of them are much better than others, meaning they also use those punches better (stronger, faster, etc). 

That is why a technique performed by a stronger shinobi would also have a stronger effect, so if the opponents are close in power, they can still use this jutsu to distract each other.