Q:

How can we show that pressure is exerted sideways too?

We can show that pressure upward and downward in a fluid is caused by weight of fluid column or volume. A simple derivation of this:
$$\text{Pressure}=\frac{\text{Force}}{\text{Area}}=\frac{\text{Weight of the fluid column}}{\text{Area}}$$
$$~~~~~=\frac{\text{mg}}{\text{A}}=\frac{\rho Vg}{\text{A}}=\frac{\rho g\times Ah}{\text{A}}=\rho g h$$
I am able to show that pressure in a fluid at a height is $\rho gh$ upwards and downwards, but how can I show that it acts sideways too?
I found a pretty reasonable answer here. So is this the correct reason?

Fluids are made of a large number of very small particles, much too
  small to see. These particles are in constant, rapid motion. They bump
  into one another. They bump into the walls of any container that holds
  them. They bump into objects in the fluid.
As the particles of a fluid bump into an object in the fluid, they
  apply forces to the object. The forces, acting over the object’s surface,
  exert pressure on the object. When the pressure in a fluid increases, the
  particles bump together more frequently. This increases the pressure
  on objects in the fluid.
The pressure a fluid exerts on an object in the fluid is applied in
  all directions. That is because the particles that make up the fluid can
  move in any direction. These particles exert forces as they bump into
  objects in the fluid.

A:

This is essentially Pascal's law. I would like to add that it depends on the fluid being in equilibrium, as well as incompressible. Otherwise local pressure 
differences can build up and the fluid can behave rather differently.
I like to look at this problem from a thermodynamics point of view. One way to think of this is that the pressure of a fluid is a measure of it's potential energy (ability to do work), and in the static case, you can almost take this to be the definition of pressure. So $U = P V$ for some element of fluid with volume $V$. Then it is a matter of mechanics to show that there exists a force $\mathbf{f} = -\boldsymbol{\nabla} U$ wherever there is a potential energy gradient. This force must be cancelled by other forces (e.g. gravity), else the fluid will move, and not be in equilibrium any more. It follows that $U$, and thus $P$, is constant in a direction perpendicular to the external force (e.g. gravity).
Now, to answer your question, if you can mentally delete the vertical walls of your fluid, then outside the fluid the potential energy of a fluid particle would be zero, so there exists a sharp energy gradient where the wall used to be. This sharp energy gradient would result in a large horizontal force. The way to balance this force, of course, is to put the walls back. I should mention that the potential energy gradient is not strictly infinite because pressures are transmitted via electrostatic forces, which are long range forces.

