Q:

Why do we need large particle accelerators?

The LHC is much larger than its predecessors, and proposed successors much larger still. Today, particle accelerators seem to be the main source of new discoveries about the fundamental nature of the world.
My lay interpretation is that particle accelerators like the LHC are essentially the only viable apparatus for performing experiments in particle physics, passive detectors of naturally energized particles notwithstanding. Experiments vary by configuration, sensors and source material, but the need for an accelerator is constant, and more powerful accelerators are able to perform experiments which are out of reach to less powerful accelerators. For the most powerful accelerators, "more powerful" seems to imply "physically larger". In these ring-shaped accelerators, for a given type of particle, its maximum power appears to be (very) roughly proportionate to circumference. I use the word "power" in a loose sense here, reflecting my loose grasp of its meaning.
Technology upgrades can make an accelerator more powerful without making it larger, e.g. the planned High Luminosity upgrade to the LHC. One imagines that an upgrade would be cheaper to build than a colossal new accelerator, yet larger accelerators are still built, so it would seem to follow that the upgrade potential of a given accelerator is limited in some way - that there is, in fact, a relationship between the size of an accelerator and its maximum power.
The first part of my question is this: what is the nature of the relationship between the size and power of a modern particle accelerator? Are there diminishing returns to the operating cost of making an accelerator more powerful? Or are there fundamental physical constraints placing a hard limit on how powerful an accelerator of a given size can be? Or is technology the main limiting factor - is it conceivable that orders-of-magnitude power increases could be efficiently achieved in a small accelerator with more advanced technology? Is it likely? 
The basic premise of these experiments seems to be that we observe the collision byproducts of energetic particles, where "energetic" presumably refers to kinetic energy, since we used an "accelerator" to energize them. To create interesting collision byproducts, the kinetic energy in the collision (measured in eV) must be at least as large as the mass of the particle (also measured in eV) we wish to create. Thus, we can observe particles of higher mass with a higher powered accelerator. 
The second part of my question is this: are particle accelerators the only way of pushing the boundaries of experimental particle physics? Is it conceivable that there is a way to produce these interesting byproducts in an experimental setting without using high-energy collisions? If not, is it conceivable that there is a way to energize particles other than by accelerating them around a track? If not, is it impossible by definition or for some physical reason? If either of these alternatives are conceivable, then assuming they're not practical replacements for large accelerators today, is it possible that they will be in the future? Is it likely? 
In a sentence, my question is this: is the future of experimental particle physics now just a matter of building larger and larger particle accelerators? 

A:

There are many competing limits on the maximum energy an accelerator like the LHC (i.e. a synchrotron, a type of circular accelerator) can reach. The main two are energy loss due to bremsstrahlung (also called synchrotron radiation in this context, but that's a much less fun name to say) and the bending power of the magnets.
The bending power of the magnets isn't that interesting. There's a maximum magnetic field that we can acquire with current technology, and the strength of it fundamentally limits how small the circle can be. Larger magnetic fields means the particles curve more and let you build a collider at higher energy with the same size. Unfortunately, superconducting magnets are limited in field: a given material has a maximum achievable field strength. You can't just make a larger one to get a larger field - you need to develop a whole new material to make them from.
Bremsstrahlung
Bremsstrahlung is German for "braking radiation." Whenever a charged particle is accelerated, it emits some radiation. For acceleration perpendicular to the path (for instance, if its traveling in a circle), the power loss is given by:
$$P=\frac{q^2 a^2\gamma^4}{6\pi\epsilon_0c^3}$$
$q$ is the charge, $a$ is the acceleration, $\gamma$ is the Lorentz factor, $\epsilon_0$ is the permittivity of free space, and $c$ is the speed of light.
In high energy, we usually simplify things by setting various constants equal to one. In those units, this is
$$ P=\frac{2\alpha a^2\gamma^4}{3}$$
This is instantaneous power loss. We're usually more interested in power loss over a whole cycle around the detector. The particles are going essentially at the speed of light, so the time to go around once is just $\frac{2\pi r}{c}$. We can simplify some more: $\gamma=\frac{E}{m}$, and $a=\frac{v^2}{r}$. All together, this gives:
$$ E_{\rm loop} = \frac{4\pi\alpha E^4}{3m^4r}$$
The main things to note from this are:

As we increase the energy, the power loss increases very quickly
Increasing the mass of the particles is very effective at decreasing the power loss
Increasing the radius of the accelerator helps, but not as much as increasing the energy hurts.

To put these numbers in perspective, if the LHC were running with electrons and positrons instead of protons, at the same energy and everything, each $6.5~\rm TeV$ electron would need to have $37\,000~\rm TeV$ of energy added per loop. All told, assuming perfect efficiency in the accelerator part, the LHC would consume about $20~\rm PW$, or about 1000 times the world's energy usage just to keep the particles in a circle (this isn't even including the actually accelerating them part). Needless to say, this is not practical. (And of course, even if we had the energy, we don't have the technology.)
Anyway, this is the main reason particle colliders need to be large: the smaller we make them, the more energy they burn just to stay on. Naturally, the cost of a collider goes up with size. So this becomes a relatively simple optimization problem: larger means higher-up front costs but lower operating costs. For any target energy, there is an optimal size that costs the least over the long run.
This is also why the LHC is a hadron collider. Protons are much heavier than electrons, and so the loss is much less. Electrons are so light that circular colliders are out of the question entirely on the energy frontier. If the next collider were to be another synchrotron, it would probably either collide protons or possibly muons.
The problem with using protons is that they're composite particles, which makes the collisions much messier than using a lepton collider. It also makes the effective energy available less than it would be for an equivalent lepton collider.
The next collider
There are several different proposals for future colliders floating around in the high-energy physics community. A sample of them follows.
One is a linear electron-positron collider. This would have allow us to make very high-precision measurements of Higgs physics, like previous experiments did for electroweak physics, and open up other precision physics as well. This collider would need to be a linear accelerator for the reasons described above. A linear accelerator has some significant downsides to it: in particular, you only have one chance to accelerate the particles, as they don't come around again. So they tend to need to be pretty long. And once you accelerate them, most of them miss each other and are lost. You don't get many chances to collide them like you do at the LHC.
Another proposal is basically "the LHC, but bigger." A $100~\rm TeV$ or so proton collider synchrotron.
One very interesting proposal is a muon collider. Muons have the advantage of being leptons, so they have clean collisions, but they are much heavier than electrons, so you can reasonably put them in a synchrotron. As an added bonus, muon collisions have a much higher chance of producing Higgs bosons than electrons do. The main difficulty here is that muons are fairly short-lived (around $2.2~\rm\mu s$), so they would need to be accelerated very quickly before they decay. But very cool, if it can be done!
The Future
If we want to explore the highest energies, there's really no way around bigger colliders:

For a fixed "strongest magnet," synchrotrons fundamentally need to be bigger to get to higher energy. And even assuming we could get magnets of unlimited strength, as we increase the energy there's a point where it's cheaper to just scrap the whole thing and build a bigger one.
Linear accelerators are limited in the energy they can reach by their size and available accelerator technology. There is research into better acceleration techniques (such as plasma wakefield accelerators), but getting them much better will require a fundamental change in the technology.

There is interesting research that can be done into precision measurements of particle physics at low energy, but for discovering new particles higher energy accelerators will probably always be desirable.

A:

The need for large sizes in conventional accelerators comes from the behavior of charged particles as a function of energy: they radiate and the higher the energy to which they are accelerated the larger the radiation, so for a given geometry and technology there is a limit. Radiation is less in linear accelerators, but there is a limit there too. The next proposal is for an international linear collider.
Larger sizes in linear and circular colliders allow for more boosting of the particles of the beam, to a point where cost effectiveness, energy losses from radiation, limits their use.
There exist alternative proposals. Back in 1977 I remember Tom Ypsilantis working very hard on a table top accelerator experiment. If you study the fields in crystals, very high electric fields exist which could accelerate muons to high energies for a given technology. (He gave up after some years and worked on the Delphi experiment on the ring imaging detector which he invented with a collaborator).
The dream of a table top accelerator is alive. Have a look here:

The team, from the U.S. Department of Energy's Lawrence Berkeley National Lab (Berkeley Lab), used a specialized petawatt laser and a charged-particle gas called plasma to get the particles up to speed. The setup is known as a laser-plasma accelerator, an emerging class of particle accelerators that physicists believe can shrink traditional, miles-long accelerators to machines that can fit on a table.
The researchers sped up the particles—electrons in this case—inside a nine-centimeter long tube of plasma. The speed corresponded to an energy of 4.25 giga-electron volts. The acceleration over such a short distance corresponds to an energy gradient 1000 times greater than traditional particle accelerators and marks a world record energy for laser-plasma accelerators.

A:

The frontiers of particle physics are at higher energies. Higher energies inevitably mean bigger machines, and yes, there is a point of diminishing returns: where the public no longer wants to foot the bill. In practical terms, we are there right now. Bigger machines could be built, but no one here wants to pay for them.
We do have higher energy sources available to us which do not involve accelerators at all, in the cosmos. The future of high-energy physics probably lies in the study of celestial objects which operate at energy levels unattainable on earth. We'll do this with bigger telescopes, and by "catching" particles that smash into our atmosphere from afar with energies that our accelerators can never match. The challenge is that those objects are very very far away from us, which makes them hard to study, but their distance is a good thing because if they weren't far away, we'd be fried to death by them.

