Why do the A320’s primary flight controls have hydromechanical backups only for the rudder? The A320 is primarily a fly-by-wire aircraft, with the pilots’ joystick and rudder-pedal inputs not going directly to the flight-control surfaces, but, rather, to the aircraft’s flight-control computers, which then decide what, if any, control-surface deflection is necessary, and send the requisite commands to the hydraulic actuators which move the control surfaces:1



However, in the event of a complete failure of the aircraft’s flight-control-computer system, the A320 can switch to a backup mode known as mechanical law, which bypasses the FCCs completely.2  In this mode, the only one of the primary flight controls to remain active is the rudder, using an emergency cable mechanism which sends the pilots’ control inputs directly to the rudder actuators; with the elevators, ailerons, and spoilerons all inactive, pitch control is maintained using manual horizontal-stabiliser trim (which uses a similar cable mechanism to send trim commands to the hydraulic motors  powering the stabiliser jackscrew) and varying engine power, and the ability to control roll separately from yaw is lost completely (to roll the aircraft in one direction or the other, it has to be yawed in the desired direction using rudder inputs and differential engine thrust; the resulting sideslip then causes the aircraft to roll towards the trailing wing, due to the positive slip-roll coupling generated by the A320’s rearward wing sweep):



The reason the elevators, ailerons, and spoilerons are unuseable in mechanical law is that, unlike the rudder, they have no direct hydromechanical connections to the cockpit controls at all, being connected only via the flight-control computers (thus leaving them cut off from the pilots’ control inputs in the event of a total failure of those computers).  If the A320 did have hydromechanical backup controls for the elevators, ailerons, and spoilerons, the pilots would be able to control the aircraft in each of its three axes separately (rather than with roll dependent on yaw, which provides very sloppy roll control with a significant time delay between the initiation of the yaw and the resulting sideslip-induced roll, and with a small but significant initial rudder-induced rolling moment in the wrong direction), and would be able to do so using the same types of control inputs as would be used in normal, fly-by-wire flight (albeit likely with considerably-increased control forces and a markedly-different force gradient), rather than having to switch to a completely different input style in mechanical law:



So why is the rudder the only one of the A320’s primary flight controls to have hydromechanical backups allowing its use in mechanical law?



1: The A320’s flight-control system, and that of all later Airbus aircraft, has several different fly-by-wire flight-control modes.  In some of these (direct law, alternate 1/2A/2B law), the pilots’ flight-control inputs in some or all control axes are simply passed straight through the computers unfiltered, with the computers parroting whatever the pilots ask of the aircraft without any of the translation present in the usual operational mode (known as normal law); however, the pilots’ inputs are still passed to the computers first, and only then to the control-surface actuators, rather than going directly aft.

2: The pitch and roll/yaw axes can each also revert to mechanical law separately in the event of a major fault in one of the primary flight controls (for instance, a double elevator jam would cause the aircraft to revert to mechanical pitch control), while still leaving the other in fly-by-wire mode.
The mechanical backup is only meant to maintain basic controllability during a temporary failure of the whole fly-by-wire system. As the OP mentioned, the rudder is deemed sufficient for basic roll control, while the horizontal stabilizer is deemed sufficient for basic pitch control even without any stability augmentation. That being said, the mechanical backup is not meant to achieve continued safe flight and landing (I would be really surprised if the mechanical backup meets all of the ARAC minimum controllability guidance); it's expected that at least one flight control computer can be rebooted to take over control after the initial failure (Ref. Airbus A320 Crew Manual). 

The reason for relegating the mechanical backup to rudder and trimmable stabilizer, instead of to elevator and ailerons, pretty much has to do with the choice of the passive sidestick as the pitch and roll inceptor. It lacks mechanical linkages to translate its motion to external assemblies, nor does it have a feel unit to provide feedback for the airloads. This leads to significant weight savings and simplicity in design, and is the primary advantage of the passive sidestick.

I suspect that the reason for even having a mechanical backup scheme is due to mistrust in FBW systems for civil aviation when they first came to use. As trust in redundancy and dissimilarity grew, newer generations of Airbus aircraft (e.g. A380, A350, A220) did away with it completely, and instead have an electrical backup servo control loop on each of the axis.