Why does reducing pressure cause the equilibrium to shift towards the side with less moles? Take the example: 

$$\ce{N2(g) + 3H2(g) &lt;-&gt; 2NH3(g)}$$

I understand that if I increase the pressure of the system, it'll shift towards the $\ce{NH3}$ side. This is because of the reaction rate increasing (increased collisions) more for the forward reaction than the reverse reaction. What I don't understand is why a decrease in pressure of the system would shift the equilibrium to $\ce{N2(g) + 3H2(g)}$. There is overall less collisions, but there's still more moles of reactants to collide and less molecules for $\ce{NH3}$ to collide and decompose.
Actually, the shift of reaction towards left on decreasing pressure (and towards right on increasing pressure) is due to Le Chatelier's Principle, which states that if a change is brought in the equilibrium conditions of a reaction, the reaction will proceed in such a manner that it counteracts the change.

In case of increasing pressure the reaction shifts to right due to lesser number of moles on right. And according to gas equation, lesser moles means lesser pressure. The opposite happens when the pressure is decreased.