Expected uptime of a cluster with unreliable components Suppose a machine has a probability $f$ of failing in an interval of $s_1$ seconds, at which point it would take $s_2$ seconds to replace. Given $d_1$ total machines, how do I compute expected fraction of the time that at least $d_2\le d_1$ machines are available?

The fraction of time any machine is available during any period of length $T$ lies between $1 - (N+1) s_2/T$ and $1 - (N-1) s_2/T$ where $N$ is the (random) number of failures occurring during the interval.  (The $\pm 1$ terms account for situations where the machine is under repair at the beginning of the interval or fails just before its end.)
We therefore need to re-interpret the single datum "probability $f$ of failing in an interval of $s_1$ seconds" in terms of the distribution of $N$ (given $T$).
One way to do this supposes failure is memoryless, whence the chance of failure in any interval of duration $s$ equals $1 - \exp(-\lambda s)$ for some parameter $\lambda.$  When this is the case, we can solve for the rate $\lambda$ as
$$\lambda = -\frac{\log(1 - f)}{s_1}$$
because that entails $1 - \exp(-\lambda s_1) = f.$
When the failure rate is $\lambda$ (per second), on average the machine is working for $1/\lambda$ seconds and not working for the next $s_2$ seconds, for a mean working time of
$$p(\lambda, s_2) = \frac{1/\lambda}{1/\lambda + s_2} = \frac{1}{1 + s_2\lambda}.$$
When the machines fail independently, the number working at any time will have a Binomial$(d_1, p(\lambda, s_2))$ distribution.  Of course, this number is (strongly) correlated among nearby times, so during any relatively short period you mustn't expect to observe exactly this distribution.  But over an extremely long duration, or if you select one random time, this distribution tells you what the chances are for any value of $d_2$ between $0$ and $d_1.$
Here is a figure of simulated machine availability according to this model.  For this simulation, $d_1=4$ machines, $\lambda = 1/50$ per second, and $s_2 = 100$ seconds for each repair.

The machine up times are color coded by machine.  The total numbers of machines in use are plotted below in black.  Here is a summary of the total amounts of time spent in each condition ($d_2 = 0, 1, \ldots, 4$) according to the simulation and the Binomial calculation.
# in use:          0   1   2  3 4
---------------- ----------------
Simulated time   199 168 167 64 7
Theoretical time 120 239 179 60 7

They differ because initially all four machines were put online simultaneously, temporarily creating an above-average count that persisted due to the temporal correlations, and this experiment was terminated only shortly after all four machines had simultaneously been down.  Over longer simulations, the results converge and agree closely.
