Q:

Where do forces point in an equilibrium system

I have the system above, with three identical balls of weight $W$ and radius $r$. The angle joining the centres is $\theta$, and the coefficient of friction between the balls and the plane is $\mu$ and between the balls $A/C$ and $B/C$ is $\mu '$. The system is in equilibrium. Denote $S$ the point of contact between balls and $P$ the point of contact between ball and plane.
My problem is that I don't know where to put the forces from which to do the calculations. Consider $A/C$: At $S$, there is a reaction force $R_S$ towards the center of both balls. But my question is, are there two forces or one more force at $S$? There are three options that I can think of:
Both a frictional force $F_{FS}$ and a force created by the weight of the ball $C$, $F_S$?
A more general force $F_S$ that includes both the mentioned above. But where would it point?
Or simply the only force that exists at $C$ is the frictional force $F_{FS}$, and the reaction force $R_S$ encompasses the force that pushes the ball $A$?

A:

Okay, the system is in equilibrium so all the forces must be balanced.
First consider the weights
We have $W$ downwards from the centre of each of the 3 balls (assuming their centres of mass are at their geometric centres).
We also have $N_A = N_B = \frac{3}{2}W$ upwards from the points of contact between the plane and balls $A$ and $B$ respectively. These are the 2 reaction forces and sum to match the weights of all 3 balls.
Now consider the points of contact between balls
At each of the 2 points of contact (between balls $A/C$ and $B/C$) you have 2 equal and opposite normal reactions $N$ which (as you correctly noted) will be directed away from the points of contact towards the centres of the balls.
Using simple trigonometry to match the weight of ball C, which only acts vertically, we see that $N$ satisfies $\frac{W}{2} = N\sin(\theta) $, which rearranges to $N = \frac{W}{2\sin(\theta)} $
Then the horizontal component of the reaction force at each point is simply $N\cos(\theta) = \frac{W\cos(\theta)}{2\sin(\theta)} = \frac{W}{2\tan(\theta)}$
Now consider the friction
Since the system is in equilibrium (i.e. unmoving) we only have to consider static friction, which ranges from $0$ up to a maximum value of $F_{max} = \mu N$.
The horizontal components of the normal reaction at the contact points between balls $A/C$ and $B/C$ are pushing balls $A$ and $B$ outwards, with a force equal to $\frac{W}{2\tan(\theta)}$ as above. To be resisted by static friction at the bases of balls $A$ and $B$, this force must be less than or equal to $F_{max} = \mu N_A = \mu N_B = \frac{3}{2}\mu W$ (assuming the coefficients of friction you supplied were the coefficients of static friction, since coefficients of kinetic friction are irrelevant to a system at rest).
This friction produces a torque on each of the balls $A$ and $B$ equal to $T = r \times F_{max} = \frac{3}{2}\mu Wr$. This requires static friction between $A/C$ and $A/B$ equal to $ F_{max}' = \frac{T}{r} = \frac{3\mu Wr}{2r} = \frac{3}{2}\mu W $ (i.e. the same friction acting at the bases of the balls, which makes sense because $r$ is the same). The static friction formula is again simply $F_{max}' = \mu' N = \frac{\mu' W}{2\sin(\theta)}$ using $N$ from above.
Hence, for the system to remain in equilibrium, we require that
$F_{max} = \frac{3}{2}\mu W \geq \frac{W}{2\tan(\theta)}$
i.e. we need $\mu \geq \frac{1}{3\tan(\theta)}$ (otherwise the plane is too slippery and the balls will slip apart)
and also we need $F_{max}' \geq F_{max} $ or $\frac{\mu' W}{2\sin(\theta)} \geq \frac{3}{2}\mu W$
i.e. we need $\mu' \geq 3\sin(\theta)\mu $ (otherwise the balls are too slippery and $C$ slips down while $A$ and $B$ roll apart).
Illustration via a special case
$\theta$ reaches its minimum possible value when balls $A$ and $B$ actually touch and their radii form an equilateral triangle of side length $2r$ and $\theta$ is 30 degrees. In this case the coefficients of static friction must satisfy
$\mu \geq \frac{1}{3\tan(30)} = \frac{\sqrt{3}}{3} \approx 0.57735 $
and 
$\mu' \geq \frac{3\sin(30)}{3\tan(30)} = \cos(30) = \frac{\sqrt{3}}{2} \approx 0.86603$

