IN addition to models using quantitative genetics ([@bib17]; [@bib1]; [@bib13]; [@bib22]), adaptive dynamics formalism has revealed diverse behaviors of evolutionary dynamics associated with continuous traits in various organisms ([@bib19]; [@bib4]; [@bib10], [@bib11]). This approach allows us to analyze the evolution of many ecological traits for which frequency-dependent selection is prevalent without specifying the derlying genetic systems controlling the trait. Even in a perfectly constant environment, a population may show intriguing temporal behavior. For example, if the trait evolves by accumulation of mutations of small magnitude and if the fitness is frequency dependent, a continuous trait may show convergence to a singular point followed by spontaneous splitting of a unimodal trait distribution into a bimodal (or multimodal) one, referred to as "evolutionary branching" ([@bib19], [@bib20]; [@bib10]). Evolutionary branching is predicted to occur at an evolutionarily singular point that is approaching stable (or convergence stable, CS) but not evolutionarily stable (ES), and it is actually observed in individual-based simulations in many models for the evolution of ecological traits.

In evolutionary branching, an evolving population spontaneously changes its trait distribution from unimodal to bimodal (or multimodal). Although evolutionary branching has been regarded as a model of sympatric speciation by some authors ([@bib5]; [@bib6]), the relationship between evolutionary branching in adaptive dynamics formalism and speciation events has been the focus of much debate (for review, see [@bib27]).

Most previous analyses of adaptive dynamics have assumed an infinite population size and that the fitness landscape acts as the deterministic driving force of evolution. However, in a finite population, stochastic fluctuations are unavoidable. For example, an individual with a lower fitness may be able to produce more offspring just by chance. Thus, a finite population can evolve against the selection gradient. As a result, stochasticity allows a population to jump from a low local peak to a higher peak. Hence, stochastic fluctuation can be an important factor in the outcome of evolution. The importance of stochasticity, or genetic drift, has been studied in detail in the context of quantitative population genetics (*e.g.*, [@bib16]; [@bib25]). On the other hand, in population genetics, random drift is known to decrease the genetic diversity of a population (*e.g.*, [@bib8]), which tends to slow down the action of natural selection. Hence, the effect of genetic drift is important to understand evolutionary dynamics, including evolutionary branching, in finite populations. In fact, [@bib3] analyzed evolutionary branching in a model including three species and reported that genetic drift tends to cause a delay in evolutionary branching. However, a more detailed study of the relationship between evolutionary branching and genetic drift is needed.

In this article, we study the effect of stochastic fluctuation caused by the finiteness of population size (random genetic drift) on evolutionary branching. As an illustrative example, we adopt a model of a public goods game studied by [@bib7], who investigated evolutionary branching by using a corresponding individual-based simulation. The game model deals with a situation that is both biologically and socially interesting: a self-organized differentiation of players into a highly cooperative group and a noncooperative group. Concerning the genetics, we in effect assume that the population is haploid and that the focal trait is controlled by a locus with many alleles. By focusing on this simplest case, we can analyze the effect of random genetic drift on the evolutionary outcome most clearly. We first show the results of individual-based simulations, using different population sizes. We show that genetic drift is especially important when the population size is small and that evolutionary branching does not necessarily occur even when the standard theory of adaptive dynamics predicts that it should. We also derive deterministic and stochastic versions of the dynamics of the trait variance. We reveal that there are two different manners in which evolutionary branching occurs: in deterministic branching, branching may occur quickly after the population reaches the singular point, while in stochastic branching, the population stays at a singularity for a period before branching out suddenly. We derive the conditions and mean branching-out times for these cases and confirm our predictions by direct computer simulations of the individual-based model.

Evolutionary Branching in a Game in a Finite Population {#s1}
=======================================================

Individual-based model {#s2}
----------------------

As a full stochastic model of a population under natural selection, mutation, and random genetic drift, we consider the following stochastic process. The population consists of *N* individuals reproducing asexually. Each individual has a genetically determined trait value $z \in \lbrack 0,1\rbrack$. At each discrete time step *t*, an individual *i* obtains a fecundity $F_{i}$ from pairwise interactions with the other *N* − 1 individuals. Each interaction is a two-player game where a payoff given to an individual *i* matched with an individual *j* is given by $f(z_{i},z_{j})$. Its fecundity $F_{i}$ is denoted by$$F_{i} = {\sum\limits_{j \neq i}{f(z_{i},z_{j})}}.$$Our framework is valid for any payoff function $f(z_{i},z_{j})$. As an example, we adopt a model by [@bib7] in which trait *z* represents the amount of investment into public goods shared by both players. In this game, the payoff function is represented by$$f(z_{i},z_{j}) = 1 + B(z_{i} + z_{j}) - C(z_{i}),$$where the benefit function is$$B(z_{1} + z_{2}) = b_{1}(z_{1} + z_{2}) + b_{2}{(z_{1} + z_{2})}^{2},$$and the cost function is$$C(z) = c_{1}z + c_{2}z^{2}.$$To keep the population size constant, the fitness of an individual with trait $z_{i}$ is denoted as$$w(z_{i}) = \frac{F_{i}}{(1/N){\sum_{j}F_{j}}}.$$

When the population size is infinitely large, [@bib7] have shown that the singular strategy $z* = (c_{1} - b_{1})/(4b_{2} - 2c_{2})$ is convergence stable when $Q_{\text{CS}} = 4b_{2} - 2c_{2} < 0$ and evolutionarily stable when $Q_{\text{ES}} = 2b_{2} - 2c_{2} < 0$. The singular strategy is referred to as an evolutionary branching point when $Q_{\text{CS}} < 0$ and $Q_{\text{ES}} > 0$. They have observed evolutionary branching in an individual-based simulation of a large population (*N* = 10,000).

In a finite population, the model includes random genetic drift, and the details of the updating rule and the order of different stages in population dynamics become important. Here, we assume a Wright--Fisher process. At each time step, fecundity is calculated for all *N* individuals. Then all individuals die and are replaced by their offspring, which are sampled multinomially, with the probability of an individual being the parent of each offspring individual proportional to the fecundity of the focal individual.

After reproduction, there is a small chance of mutation μ, and the mutant has a trait value slightly different from that of the parent. To be specific, it follows a normal distribution with a mean equal to the parent's and a variance $\sigma^{2}$. Mutants are constrained to be between 0 and 1, which confines trait *z* within an interval \[0, 1\].

Evolutionary branching depends on the population size {#s3}
-----------------------------------------------------

All simulations were run with parameter values for which a unique singular point exists that is convergence stable and evolutionarily nonstable (*i.e.*, the evolutionary branching point). Thus, evolutionary branching is expected when the population size is infinitely large.

When the population size is large (*N* = 8000), evolutionary branching occurs as soon as the trait evolves to reach the CS value *z*\* ([Figure 1A](#fig1){ref-type="fig"}). For an intermediate population size (*N* = 1000), the population lingers around the CS value, *z*\*, before it finally shows evolutionary branching. The waiting time until the evolutionary branching varies considerably among different simulation runs with the same parameter values ([Figure 1, B and C](#fig1){ref-type="fig"}). For a small population size (*N* = 200), we do not observe evolutionary branching and the population seems to stay around *z*\*, even though *z*\* is not evolutionarily stable ([Figure 1D](#fig1){ref-type="fig"}) (see also [@bib26]).

![Evolutionary dynamics. (A) A simulation run with *N* = 8000. (B and C) Two different simulation runs with *N* = 1000. (D) A simulation run with *N* = 200. Parameters: $\mu = 0.01,\,\sigma = 0.02,\,(b_{1},b_{2},c_{1},c_{2}) = (6.0, - 1.4,\, 4.56, - 1.6)$.](229fig1){#fig1}

The evolutionary branching dynamics are characterized by an abrupt increase or explosion of the variance of the focal phenotype (the trait variance). If all the parameters for the interactions are the same, a small population does not branch, an intermediately large population shows delayed branching, and a large population branches immediately. The intuitive reason for the smallness of the population size to suppress evolutionary branching is that genetic diversity is reduced by genetic drift every generation. In the following sections, we provide approximate dynamics for the trait variance to understand these simulation results.

Dynamics of Trait Variance {#s4}
==========================

The full (individual-based) model is analytically intractable. Therefore, we derive a reduced simplified dynamical system here, which is tractable and gives predictions on the behavior of the full model. One generation in the reduced model consists of two substeps.

Substep 1: Natural selection and mutation {#s5}
-----------------------------------------

Let $\phi(z)$ denote a distribution of trait *z*, which is normalized to satisfy $\int{\phi(z)dz = 1}$. When a fitness function $w(z)$ is given and selection and mutation are sufficiently weak, the trait distribution in the offspring pool after mutation, denoted by $\chi(z)$, can be approximated by the following replicator equation plus a mutation term,$$\chi(z) = \phi(z) + \left\lbrack {w(z) - \overset{-}{w}} \right\rbrack\phi(z) + \frac{\mu\sigma^{2}}{2}\frac{\partial^{2}}{\partial z^{2}}\phi(z),$$where $\overset{-}{w}$ is the population average of fitness values. The second term on the right-hand side represents the effect of selection and the third term represents the effect of mutation. The Taylor expansion of the fitness function around the average trait value $\overset{-}{z}$ is denoted as$$w(z) = w(\overset{-}{z}) + w_{1}(z - \overset{-}{z}) + \frac{w_{2}}{2}{(z - \overset{-}{z})}^{2},$$where the coefficients $w_{1}$ and $w_{2}$ are determined by the current state $\overset{-}{z}$. The coefficient $w_{1}$ represents the selection gradient. The coefficient $w_{2}$ indicates stabilizing selection if it is negative and indicates disruptive selection if it is positive. The derivation of $w_{2}$ from the payoff function $f(z_{i},z_{j})$ is explained in *Appendix A*. For parameter values used in our simulations, we have $w_{2} \approx 0.1$, implying that disruptive selection is operating. The population average and the variance of trait values in the offspring trait distribution $\chi(z)$ are respectively denoted as$$\text{Av}_{\chi}\lbrack z\rbrack = \overset{-}{z} + w_{1}m_{2} + \frac{w_{2}}{2}m_{3},$$$$\text{Var}_{\chi}\lbrack z\rbrack = m_{2} - ({w_{1}}^{2} + \frac{w_{2}}{2}){m_{2}}^{2} + w_{1}m_{3} + \frac{w_{2}}{2}m_{4} + \mu\sigma^{2},$$where $m_{k}$ denotes the *k*th centralized moment of $\phi(z)$. The above approximation is derived by neglecting $m_{5}$ and higher-order moments (see *Appendix B*).

Substep 2: Random sampling (or random genetic drift) {#s6}
----------------------------------------------------

From the offspring pool $\chi(z)$, *N* individuals are independently sampled to become adults. Hence, even for a fixed $\chi(z)$, different distributions of adult phenotypes are realized according to a probability distribution (probability measure) over all possible trait distributions. After sampling *N* individuals independently from $\chi(z)$, the expectation of the average trait value is identical to the average trait value in $\chi(z)$. In contrast, the expectation of the trait variance is decreased by a factor (*N* − 1)/*N* ([@bib8]). Hence, we have the following:$$\begin{array}{l}
{E\lbrack\text{Av}\lbrack Z\rbrack\rbrack = \text{Av}_{\chi}\lbrack z\rbrack} \\
{E\lbrack\text{Var}\lbrack Z\rbrack\rbrack = \frac{N - 1}{N}\text{Var}_{\chi}\lbrack z\rbrack.} \\
\end{array}$$

Combining the two substeps, the expected changes in the population average and the variance in one generation are given by$$E\lbrack\Delta\text{Av}\lbrack Z\rbrack\rbrack = w_{1}m_{2} + \frac{w_{2}}{2}m_{3},$$$$\begin{array}{l}
{E\lbrack\Delta\text{Var}\lbrack Z\rbrack\rbrack} \\
{= \frac{N - 1}{N}\left\lbrack {m_{2} + w_{1}m_{3} - {w_{1}}^{2}{m_{2}}^{2} + \frac{w_{2}}{2}(m_{4} - {m_{2}}^{2}) + \mu\sigma^{2}} \right\rbrack - m_{2}.} \\
\end{array}$$We are particularly interested in the dynamics of the variance of the trait distribution. To analyze Equation 8 further, we here assume that the trait distribution is close to a normal distribution whose mean and variance obey the dynamics given by Equation 8. For normal distributions, $m_{3} = 0$ and $m_{4} = 3{m_{2}}^{2}$. Using these relationships, we have the following recursive equations for the mean and variance of the trait.:$$\Delta\overset{-}{z} = w_{1}m_{2}$$$$\Delta m_{2} = \frac{N - 1}{N}(w_{2} - {w_{1}}^{2}){m_{2}}^{2} - \frac{1}{N}m_{2} + \frac{N - 1}{N}\mu\sigma^{2}.$$The first, second, and third terms on the right-hand side of Equation 9b represent the effect of natural selection, the loss of diversity due to genetic drift, and the increase of diversity due to mutation, respectively. Without selection, the variance $m_{2}$ converges to $\left( {N - 1} \right)\mu\sigma^{2}$.

Condition for the Explosion of Trait Variance {#s7}
=============================================

As long as the selection gradient does not vanish ($w_{1} \neq 0$), the mean trait value changes according to Equation 9a, which is a standard result of population genetics ([@bib8]) and corresponds to the canonical equation in adaptive dynamics theory ([@bib4]; [@bib2]). If a singular point *z*\*, where the selection gradient vanishes ($w_{1} = 0$), is convergence stable, then the system approaches the singular point with time ([@bib10]). Once the mean trait (*i.e.*, the first moment of trait distribution) reaches the singularity, it will not change any longer. However, the trait variance (*i.e.*, the second centralized moment of the distribution) may change with time. See [Figure 2](#fig2){ref-type="fig"} for a schematic illustration. Here we focus on the recursion for the trait variance,$$\Delta m_{2} = \frac{N - 1}{N}w_{2}{m_{2}}^{2} - \frac{1}{N}m_{2} + \frac{N - 1}{N}\mu\sigma^{2},$$which indicates that the one-generational change in the trait variance is equal to the sum of three terms: the natural selection term, the genetic drift term, and the mutation term. The right-hand side of Equation 10 represents the "force" acting on the trait variance ([Figure 3](#fig3){ref-type="fig"}). Note that the orders of the three terms in Equation 10 are ${m_{2}}^{2}$, $m_{2}$, and 1, respectively. When the variance is small, the mutation (the third term) is dominant, and it increases the trait variance. As the variance increases, the genetic drift (the second term) increases its magnitude and it reduces the trait variance. The balance between these two processes would result in the equilibrium $m_{2}* \approx N\mu\sigma^{2}$, which represents the variance determined by mutation--drift balance. When the trait variance is large, selection (the first term) becomes dominant. Selection decreases the trait variance when $w_{2} < 0$ (stabilizing selection), but it increases the trait variance when $w_{2} > 0$ (disruptive selection).

![A schematic illustration of phenotype distribution $\phi(z)$ and fitness function $w(z)$ when the mean trait value $\overset{-}{z}$ has reached the singular strategy *z\**. The trait variance is $\text{Var}\lbrack\phi(z)\rbrack = m_{2}$ so the standard deviation is $\sqrt{m_{2}}$.](229fig2){#fig2}

!["Force" plotted against the trait variance based on Equation 10. Dotted, solid, and thick solid curves represent stabilizing selection ($w_{2}$ = −0.2), weak disruptive selection ($w_{2}$= 0.2), and strong disruptive selection ($w_{2}$ = 0.6), respectively. They also correspond to cases i, ii, and iii in the main text. Parameters: *N* = 400, $\mu = 0.01,\,\sigma = 0.02$.](229fig3){#fig3}

Combining these, we have three different cases, depending on the parameters.

1.  Case i: When $w_{2} < 0$, stabilizing selection is at work. Then the dynamics of the trait variance (Equation10) have a single positive equilibrium$$m_{2}* = \frac{1 + \sqrt{1 - 4{(N - 1)}^{2}w_{2}\,\mu\sigma^{2}}}{2(N - 1)w_{2}},$$which is globally stable, and the trait variance converges to this value for any initial value. At this equilibrium, the trait variance is maintained by mutation, which increases variance, and by genetic drift and stabilizing selection, which decrease variance. Especially when the stabilizing selection is weak, trait variance converges to the level maintained by mutation--drift balance ($m_{2}* \approx \left( {N - 1} \right)\mu\sigma^{2}$) when $\left| w_{2} \right|$ is small.

2.  Case ii: When $0 < w_{2} < 1/4\mu\sigma^{2}{(N - 1)}^{2}$, disruptive selection is at work, but it is not very strong. Then the dynamics of the trait variance have two positive equilibria: the smaller one is stable, but the larger one is unstable. If the trait variance starts from a small value, it converges to the smaller equilibrium, where the trait variance is maintained mainly by the mutation--drift balance. If the trait variance starts from a larger value, then disruptive selection dominates, leading to the unlimited growth of the trait variance.

3.  Case iii: When $w_{2} > 1/4\mu\sigma^{2}{(N - 1)}^{2}$, disruptive selection is strong. The dynamics of the trait variance no longer have a stable equilibrium. Starting from any initial value, the trait variance keeps increasing without limit.

[Figure 4](#fig4){ref-type="fig"} shows a bifurcation diagram of the variance dynamics given by Equation 10, with the stabilizing/disruptive selection coefficient ($w_{2}$) as a bifurcation parameter. [Figure 5](#fig5){ref-type="fig"} shows a bifurcation diagram in which the population size (*N*) is a bifurcation parameter. We have a similar bifurcation diagram where mutational effects ($\mu\sigma^{2}$) serve as a bifurcation parameter (not shown). We regard the explosion of the trait variance in the dynamics of Equation 10 as evolutionary branching. Then, these bifurcation diagrams show that evolutionary branching occurs as a result of a saddle-node bifurcation. Evolutionary branching is predicted to occur at a non-ES singular point when disruptive selection intensity, population size, mutation rate, and mutation step size are sufficiently large. Specifically, the deterministic dynamics of Equation 10 predict that evolutionary branching occurs when $4\mu\sigma^{2}{(N - 1)}^{2}w_{2} > 1$ holds.

![Stable (solid) and unstable (dashed) branches of the trait variance $m_{2}$ with disruptive selection intensity $w_{2}$ as a bifurcation parameter. Arrows indicate the direction of systematic change in $m_{2}$. Parameters: *N* = 1000, $\mu = 0.01,\,\sigma = 0.02$.](229fig4){#fig4}

![Stable (solid) and unstable (dashed) branches of the trait variance $m_{2}$ with population size *N* as a bifurcation parameter. A Log plot is shown. Arrows indicate the direction of systematic change in $m_{2}$. Parameters: $\mu = 0.01,\,\sigma = 0.02,\,(b_{1},b_{2},c_{1},c_{2}) = (6.0, - 1.4,\, 4.56, - 1.6)$.](229fig5){#fig5}

Stochastic Calculation of the Explosion of Variance {#s8}
===================================================

The deterministic dynamics of the trait variance represented by Equation 10 indicate only the expectation of one-generational change. There must be stochasticity caused by the genetic drift. Results of an individual-based model, such as those illustrated in [Figure 1](#fig1){ref-type="fig"}, clearly show stochasticity. To account for stochasticity, the deterministic difference of Equation 10 needs to be modified by an additional term. In this section, we derive the stochastic differential equation model considering the stochasticity of trait variance dynamics.

The deterministic dynamics of Equation 10 can be approximated by the ordinary differential equation (ODE)$$\frac{d}{dt}m_{2} = f(m_{2}),$$where$$f(x) = \frac{(N - 1)w_{2}x^{2} - x + (N - 1)\mu\sigma^{2}}{N}.$$If stochasticity has no correlation over generations, and if the paths can be approximated as continuous, then we can incorporate the Brownian motion term, and the differential Equation 11 becomes the stochastic differential equation (SDE)$$dM_{2} = f(M_{2})dt + \sqrt{\frac{2}{N}}M_{2}dB,$$where *dB* denotes Brownian motion and $M_{2}$ denotes a random variable representing the trait variance. $f(M_{2})$ is the systematic change per generation in variable $M_{2}$ and is given by Equation 11b.

The second term on the right-hand side of Equation 12 represents stochasticity in trait variance caused by random genetic drift (see *Appendix C* for derivation). The stochastic fluctuation term depends on the trait variance itself (geometric Brownian motion), because the amplitude of fluctuation in the trait variance in each generation of offspring is large when the trait variance in each parental generation is large. The stochastic term also depends on population size, because fluctuation is averaged out in large populations. Hence, random genetic drift becomes less important in larger populations. The system is similar to but different from the motion of a particle moved by force shown in [Figure 3](#fig3){ref-type="fig"} under heat noise.

Stationary distribution {#s9}
-----------------------

The corresponding Fokker--Planck equation (or forward Kolmogorov equation) to Equation 12 is given by$$\frac{\partial}{\partial t}p(m_{2};t) = - \frac{\partial}{\partial m_{2}}\left\lbrack {f(m_{2})p(m_{2};t)} \right\rbrack + \frac{1}{2}\frac{\partial^{2}}{\partial{m_{2}}^{2}}\left\lbrack {\frac{2}{N}{m_{2}}^{2}p(m_{2};t)} \right\rbrack$$([@bib9]), where $p\left( {m_{2};t} \right)dm_{2} = \Pr\left\lbrack {m_{2} \leq M_{2}\left( t \right) < m_{2} + dm_{2}} \right\rbrack$ is the probability density for the trait variance $m_{2}$. The stationary distribution of the trait variance is$$\widetilde{p}(x) = C\, x^{- 3}e^{(N - 1)a(x)},$$where *C* is a normalizing constant and$$a(x) = \frac{w_{2}x^{2} - \mu\sigma^{2}}{x}.$$[Figure 6](#fig6){ref-type="fig"} illustrates an example of stationary distribution of the trait variance, showing that the system tends to show slightly smaller trait variance than the locally stable value in Equation 11, because the fluctuation is weaker when the variance is smaller (geometric Brownian motion).

![The stationary distribution $\widetilde{p}(m_{2})$ of the trait variance $m_{2}$ (Equation 14). A Log-Log plot is shown, and thus the probability density is much higher at the peak than it appears. The locally stable value of the variance according to Equation 11 is $m_{2}* \approx$ 0.00081, which is close to $N\mu\sigma^{2}$ = 0.0008 (dashed line). Note that a local peak lies at a value \<0.0008 because of geometric Brownian motion. Parameters: *N* = 200, $\mu = 0.01,\,\sigma = 0.02,\,(b_{1},b_{2},c_{1},c_{2}) = (6.0, - 1.4,\, 4.56, - 1.6)$.](229fig6){#fig6}

### Comparison with the results of simulations: {#s10}

We compared the stationary distribution of the trait variance between the theoretical prediction given by Equation 14 and the observed distribution obtained from the individual-based simulation (*Evolutionary Branching in a Game in a Finite* *Population* section). [Figure 7](#fig7){ref-type="fig"} shows good agreement between the diffusion approximation (*i.e.*, the SDE model) and the corresponding simulation result with a small population size. In particular, the approximation captures the shift of the mode---the peak of the distribution lies at the value smaller than $N\mu\sigma^{2} = 0.0008$. The average trait variance in an interval shown in [Figure 7](#fig7){ref-type="fig"} is close to $N\mu\sigma^{2}$ for both analytic prediction (Equation 14) and simulation results.

![The stationary distribution of the trait variance $m_{2}$. Curves represent an analytic result (Equation 14), while bars are a histogram of realized trait variances in a simulation run for $10^{6}$ time steps. The dashed line represents $N\mu\sigma^{2}$ = 0.0008. In the simulation, the initial trait distribution was set as monomorphic at *z* = 0.2. The transition from this initial state to the stationary state was negligible as it took only ∼5000 time steps. Parameters: *N* = 200, $\mu = 0.01,\,\sigma = 0.02,\,(b_{1},b_{2},c_{1},c_{2}) = (6.0, - 1.4,\, 4.56, - 1.6)$.](229fig7){#fig7}

Waiting time for evolutionary branching {#s11}
---------------------------------------

In case ii, there are stable and unstable equilibria for the trait variance, the latter being larger than the former. If the initial value of trait variance is smaller than the unstable equilibrium, the deterministic model Equation 11 predicts that variance would be maintained indefinitely. In contrast, if the initial value is larger than the unstable equilibrium, it quickly increases and grows without limit. Hence, whether trait variance eventually explodes depends on the initial condition. In the presence of stochasticity, however, the trait variance would not be maintained at smaller values than the unstable equilibrium; instead, it would fluctuate around the stable equilibrium for a period, but then eventually increase beyond the unstable equilibrium and diverge to infinity. Hence, under the stochastic differential equation, the trait variance is predicted to explode eventually even if the initial value is small. The timing of this explosion is controlled by the stochasticity.

The mean waiting time prior to explosion is a typical "first passage time" problem in diffusion theory. This has also been used in calculating the mean time of fixation in population genetics (Kimura and Ohta 1968) and the mean time to extinction in conservation biology (*e.g.*, [@bib18]; [@bib12]) and physiology (*e.g.*, [@bib23]).

To study the expectation of waiting time, we assume that a system has a reflecting boundary at $m_{2} = l \approx 0$ and an absorbing boundary at $m_{2} = r$ (note that the left boundary can never be achieved because the deterministic force is increasing the variance while stochastic fluctuation vanishes at $m_{2} = 0$; *i.e.*, Equation 12 is singular at $m_{2} = 0$). Let $T(\eta)$ be the expected waiting time until the system reaches the right absorbing boundary $m_{2} = r$ when the initial value of the trait variance is $m_{2}(0) = \eta$. To obtain $T(\eta)$, we use the backward Kolmogorov equation$$- 1 = f(\eta)\frac{d}{d\eta}T(\eta) + \frac{\eta^{2}}{N}\frac{d^{2}}{d\eta^{2}}T(\eta),$$with the boundary conditions $T(r) = 0$ and $T^{\prime}(l) = 0$ (see *Appendix E*). The solution is$$T(\eta) = N{\int_{\eta}^{r}{{\int_{l}^{y}{x^{- 3}ye^{(N - 1)(a(x) - a(y))}dx}}dy}},$$where $a(x)$ is given by Equation 14b. The expected waiting time is smaller when the population size is larger, disruptive selection is stronger, or mutational effects are larger (see *Appendix D*).

### Comparison with the results of simulations: {#s12}

We compared the mean waiting times observed in the computer simulations of the individual-based model and the prediction by the approximate stochastic differential equation model given by Equation 16. Once the system reaches a convergence stable point, the trait variance $m_{2}$ is expected to be close to the locally stable value $N\mu\sigma^{2}$ (see also our results on stationary distribution). Thus, we numerically calculate $T(N\mu\sigma^{2})$ and compare the corresponding result in individual-based simulation ([Figure 8](#fig8){ref-type="fig"}). We also derive the probability distribution of the first passage time by using Kolmogorov's backward equation in *Appendix E*, but the fit of the shape of the probability distribution of waiting times to the distribution observed in individual-based simulations is less than perfect ([Figure 9](#fig9){ref-type="fig"}).

![A Log plot of waiting times for evolutionary branching for different population sizes (*N*). The thick and thin curves represent analytic results based on a SDE model (Equation 16) and those based on an ODE model (Equation 11), respectively. Each open circle represents the result of a single run in individual-based simulation. The first time the trait variance exceeded 0.1 is shown. In the simulation, the initial trait distribution was set as monomorphic at *z* = 0.2 and each run consisted of $10^{6}$ time steps. Simulation runs in which the variance did not explode are plotted at $10^{6}$. Time required to reach a singular point in the simulation was ∼5000 when *N* = 200 and ∼2000 when *N* = 1800, which is included to show open circles. Thus, open circles are expected to lie slightly above the curve. Parameters: $\mu = 0.01,\,\sigma = 0.02,\,(b_{1},b_{2},c_{1},c_{2}) = (6.0,\, - 1.4,\, 4.56,\, - 1.6)$.](229fig8){#fig8}

![The distribution of waiting times. Curves represent an analytic result (see *Appendix E*), while bars represent a histogram of the waiting time obtained in 1000 different runs of individual-based simulation. Parameters: *N* = 1000, $\mu = 0.01,\,\sigma = 0.02,\,(b_{1},b_{2},c_{1},c_{2}) = (6.0,\, - 1.4,\, 4.56,\, - 1.6)$.](229fig9){#fig9}

As we decrease *N*, the waiting time increases exponentially, and it eventually becomes impossible to escape from the locally stable unimodal state in a reasonable time ([Figure 8](#fig8){ref-type="fig"}). This implies that evolutionary branching is unlikely to occur in a small population.

When *N* is very large, genetic drift can be neglected and the system is approximated by the ODE: $dm_{2}/dt = w_{2}{m_{2}}^{2} + \mu\sigma^{2}$. Thus, the waiting time can be approximated by the time required to move from $m_{2} = N\mu\sigma^{2}$ to the right boundary $m_{2} = r$. This time is ∼1500 for *N* = 1000 and ∼960 for *N* = 2000. This calculation, in conjunction with [Figure 8](#fig8){ref-type="fig"}, shows that the stochastic fluctuation plays a dominant role even in a population of 1000 individuals.

Our reduced model measures genetic diversity in trait distribution. An alternative measure of genetic diversity is the number of segregating alleles, which depends only on μ and is predicted by Ewens' sampling formula when selection is absent ([@bib8]). [Figure 10](#fig10){ref-type="fig"} shows the effect of mutation rate μ and mutation step size $\sigma^{2}$. Simulations confirmed our prediction that only the value of $\mu\sigma^{2}$ matters, even though the numbers of segregating alleles in the simulation with *N* = 1800 were ∼50 and ∼150 for $\mu = 0.0025$ and 0.01, respectively. It is also confirmed that smaller mutational effects require a longer waiting time until evolutionary branching.

![A Log plot of waiting times for evolutionary branching for different population sizes and mutation parameters. The top panel ($\mu = 0.0025,\sigma = 0.02$) depicts a result for a smaller mutation rate than in [Figure 8](#fig8){ref-type="fig"}, while the bottom panel ($\mu = 0.01,\sigma = 0.01$) depicts a result for a smaller mutation step size. As $\mu\sigma^{2} = 10^{- 6}$ is the same, the analytic prediction curves are the same. The two simulation results are also undistinguishable. Parameters: $(b_{1},b_{2},c_{1},c_{2}) = (6.0, - 1.4,\, 4.56, - 1.6)$.](229fig10){#fig10}

Discussion {#s13}
==========

In this article, we have studied the effect of stochastic fluctuation due to finite population size on evolutionary branching. We have derived an ordinary differential equation and then a stochastic differential equation for the dynamics of trait variance. The analysis has revealed that there are two different manners in which evolutionary branching occurs: deterministic branching and stochastic branching. In the case of deterministic branching, evolutionary branching occurs relatively soon after the mean trait reaches the branching point. This occurs when disruptive selection intensity $w_{2}$ exceeds a threshold value, $w_{2} > 1/(4\mu\sigma^{2}{(N - 1)}^{2})$, which can be approximated by ${1/N} < 2\sqrt{w_{2}\mu\sigma^{2}}$. This condition for deterministic branching implies that the intensity of random genetic drift $1/N$ is smaller than disruptive selection intensity $w_{2}$ and the mutational effects $\mu\sigma^{2}$ (see later in article). If the opposite inequality holds, *i.e.*, when disruptive selection is weaker than the threshold, then a unimodal distribution is locally stable in the deterministic model and stochastic fluctuation is necessary for the population to undergo evolutionary branching. We call this latter type of evolutionary branching "stochastic branching." We observed a relatively long waiting time until evolutionary branching occurs in simulations. The distribution and expectation of the waiting time were mathematically calculated and these predictions were consistent with computer simulations of the individual-based model. Considering the several assumptions we have adopted, the consistency is better than expected, although not perfect. As population size or mutational effect decreases, the waiting time exponentially increases. Mathematically, this is because these factors appear in the exponent in the expression of the expected waiting time (Equation 16). Although the increase of the waiting time is a continuous phenomenon resulting from the underlying stochasticity in individual-based simulations, we can understand evolutionary branching by classifying it as either deterministic branching in which stochastic fluctuation is unnecessary or stochastic branching in which stochastic fluctuation is necessary.

Our analysis has revealed that evolutionary branching is almost impossible in some biological situations in which the effective population size or mutational effect is sufficiently small. As a population size increases, our condition for deterministic evolutionary branching approaches and converges to the standard branching-point condition. The condition can be written as $4w_{2}N^{2}\mu\sigma^{2} > 1$, which can be used as an approximate indicator; for example, a population with size *N* = $10^{8}$ with mutational effects $\mu\sigma^{2} = 10^{- 14}$ is predicted to show deterministic branching when disruptive selection intensity is $w_{2} = O(1)$, although such individual-based simulation is almost impossible.

The condition separating deterministic and stochastic branching is understood intuitively by considering the force suppressing variance (genetic drift) and the two forces promoting the variance (disruptive selection and mutation). The balance between them determines which type of branching occurs. The condition for deterministic branching, ${1/N} < 2\sqrt{w_{2}\mu\sigma^{2}}$ can be rewritten as ${(1/N)}^{2} < 4w_{2} \cdot \mu\sigma^{2}$, which means$$\left( \text{genetic}\,\text{drift} \right)^{2} < 4 \times \left( \text{selection}\,\text{effect} \right) \times \left( \text{mutation}\,\text{effect} \right).$$This implies that deterministic branching occurs when the intensity of genetic drift is smaller than double the geometric average of the selection and mutation intensities. It is important to note that the condition is determined by $N^{2}$ instead of *N*.

To mimic the full individual-based model by using the deterministic and stochastic equations of the trait variance, we have adopted several simplifying assumptions in deriving the dynamics: *i.e.*, trait distribution is assumed to be close to normal, and the trait variance is small enough to justify series expansion. From simulation results, it is clear that the trait distribution finally becomes multimodal when evolutionary branching occurs. Thus, our analytic results are valid only for the initial growth of the trait variance where unimodal distribution (approximated by a normal distribution) becomes broader at the singular point. This initial explosion is followed by highly nonlinear and complex dynamics that realize a multimodal distribution. Nevertheless, the agreement of our analytic prediction and simulation results suggests that our reduced model captures the onset of evolutionary branching.

Based on individual-based simulations, [@bib3] reported delayed evolutionary branching and proposed two reasons: random genetic drift and extinction of incipient branches. In contrast to [@bib3], our model has no fluctuation in the total population size. Thus, the delayed or absent evolutionary branching is caused by demographic stochasticity---the fluctuation in the number of offspring each individual produces. Our simple model has enabled us to derive the condition for evolutionary branching mathematically, with random genetic drift taken into consideration, which has been only verbally proposed before. In addition, our mathematical analysis and simulation results have revealed that the mutation rate and step size are as important as population size.

In this article, to illustrate the effect of genetic drift on the evolutionary branching, we used as an example a pairwise interaction game that incorporated a Wright--Fisher update. We conjecture that the extension of our model to include a Moran update or to a multiplayer game is readily possible. In the case of including a Moran update, the loss of genetic diversity (Equation 7) should be appropriately modified and also the unit of time should be rescaled since *N* Moran steps correspond to a single Wright--Fisher step. The result of our model should be easily extended to resource-competition models, which have been intensively studied in adaptive dynamics (*e.g.*, [@bib5]; [@bib24]; [@bib21]).

We have focused on the dynamic processes in which the population escapes from the locally stable value of the trait variance, while keeping its trait distribution as a unimodal distribution. Conceptually, if we allow all possible trait distributions, we might have two locally stable states, one of which corresponds to a unimodal state and a second that corresponds to a fully branched state. Stochastic branching in the present model considers only the case in which a system located at the unimodal state escapes from this local optimum by stochastic fluctuation (random genetic drift). However, a branched state can also evolve back to the unimodal state by genetic drift; thus, a stochastic analysis including both escaping from and evolving back to the unimodal distribution would give us an even deeper understanding of evolutionary branching.

We thank the following people for their very helpful comments: C. Hauert, Y. Kobayashi, C. Miura, W. Nakahashi, H. Ohtsuki, and A. Sasaki. This work was supported by Japan Science and Technology Precursory Research for Embryonic Science and Technology (J.Y.W.), by a Grant-in-Aid for Scientific Research (B) from Japan Society for Promotion of Science (to Y.I.), and also by the support of the Environment Research and Technology Development Fund (S9) of the Ministry of the Environment, Japan.

Communicating editor: W. Stephan
