This is a fun question and has about 4 or 5 different factors at play:
Types of forces involved: Pressure vs. Inertia Types of resistance involved: Rigidity vs. Plasticity Objects involved: Soldiers vs. Buildings
Scenario: Damage from a bomb blast (energy wave) vs. shrapnel impact/penetration (kinetic collision).
How damage is applied with:
1. Blast - the PSI of a blast is uniform over the face of it's wave-form (sphere). All material objects are subject to this energy being propagated over their surface and structure.
2. Shrapnel/Bullet - the kinetic energy (m*V) is imparted during a collision. There is very high relative force at the point of impact, at which point the total energy is propagated throughout the object.
The damage of either occurs when the strength of the medium is unable to absorb or deflect the energy, and thus the material bonds are broken. 621
In buildings this results in cracks or holes in lesser cases, or structural collapse in greater ones. For soldiers there are additional physiological factors that relate to perforations of internal organs, or limbs torn off, or the "fine-red-mist" scenario in higher energy exchanges.
If the tensile strength of a surface is greater than the force of a collision it will be mostly reflected - so buildings hit by shrapnel are much less likely to be damaged as the actual force is relatively low.
While most shrapnel is very small, it travels very fast. It is easily able to penetrate the surface resistance and impart its energy on the structure. At this point we can see that soldiers cannot resist nearly as much total force as a building.
Now for some figures: A 9mm/.40 cal bullet has about 350 to 400 ft lbs. of energy. This converts to 2.4 - 2.7 PSI, however this is applied over a very small point (probably not even a square inch). At 5 PSI the force is approximately equivalent to .357 magnum, or .45 ACP (standard officer's side arm). You won't be knocking down a building with one of those, yet they do a good job of stopping a soldier.
Why does a building not withstand a 5 psi blast then? Find the surface area of the building in square inches (an 8' x 12' wall is 1152 square inches), multiply by 5 and then do the same for the human (approximately 250 square inches).
Optionally divide the 250 / 1152 and see that the human is absorbing only 20% of the damage. Factor in the reduction due to deformation of the surface (I have no idea what figures would be in play here.), and humans suddenly look like super-men vs. blast damage. 441