Ansari-Bradley test I am trying to understand Ansary-Bradley test variance. Suppose n+m is even number then I know that $E(X)=  \frac{n(n+m+2)}{4}$ and that $Var(X)= \frac{nm(n+m+2)(n+m-2)}{48(n+m-1)}$. So I want to show that $ Var(X)= E(X^2)-E(X)^2=E(X^2)- \left(\frac{n(n+m+2)}{4}\right)^2=\frac{nm(n+m+2)(n+m-2)}{48(n+m-1)}$. However I don't understand how can I find $E(X^2)$?

The Ansari-Bradley (A-B) test statistic $\mathscr X$ is the sum of values in a random sample $X$ (without replacement) of size $n$ from a population $\Omega$ of $n+m$ (not necessarily distinct) numbers.
Some general observations about the variances of such samples will be helpful.

*

*We may center the population at zero because that does not change its variance and therefore will not change the variance of the sum of the sample.  In other words, by subtracting a constant from all the numbers in the population we may assume $$0 = \sum_{\omega\in\Omega}\omega.$$


*$X$ determines $n+m$ random variables, one for each outcome $\omega\in\Omega,$ via $I(\omega) = 1$ when $\omega\in X$ and $I(\omega) = 0$ otherwise.  Therefore $$\mathscr X = \sum_{x\in X} x = \sum_{\omega\in\Omega} \omega I(\omega).$$


*Because $X$ is a random sample without replacement, the $I(\omega)$ are exchangeable.  In particular, they share the same mean $\mu$, variance $\sigma^2 = \mu(1-\mu)$ (because the $I(\omega)$ are Bernoulli variables) and covariances $\rho\sigma^2.$  We will need to find $\rho.$


*Because the sample size $n$ is fixed, the sum of all the $I(\omega)$ is $n$ and therefore the expectation of that sum is $n$ and its variance is zero:
$$\begin{aligned}
n &= E\left[\sum_{\omega\in\Omega}I(\omega)\right] = (n+m)\mu;\\
0 &= \operatorname{Var}\left(\sum_{\omega\in\Omega}I(\omega)\right) = (n+m)\left(1 + (n+m-1)\rho\right)\sigma^2.\end{aligned}$$
This system of equations (which, being separately linear in $\mu$ and $\rho,$ is easy to solve) has the unique solution $$\mu = \frac{n}{n+m};\ \rho = -\frac{1}{n+m-1};\ \sigma^2 = \mu(1-\mu) = \frac{nm}{(n+m)^2}.$$


*From (1), (2), and (4),
$$\begin{aligned}
\operatorname{Var}(\mathscr X) &= \sum_{\omega\in\Omega} \omega^2 \sigma^2 + \sum_{\omega\ne\eta\in\Omega} \omega\eta\, \rho \sigma^2 \\
&= \sigma^2(1-\rho)\sum_{\omega\in\Omega} \omega^2  + \sigma^2\rho\left(\sum_{\omega\in\Omega} \omega\right)^2 \\
&= \sigma^2(1-\rho)\sum_{\omega\in\Omega} \omega^2\\&= \frac{mn}{(n+m)(n+m-1)}\sum_{\omega\in\Omega} \omega^2.
\end{aligned}$$

In the A-B test, $n$ is the count of one batch of data and $m$ is the count of the other batch to which it is being compared.  When all $n+m$ values are combined, let the "min rank" of any value be its count from the nearest end: that is, the largest and smallest have min ranks of 1; the second largest and second smallest have min ranks of 2; and so on.
When the total $n+m$ is even and there are no tied values, the min ranks corresponding to the sorted values go from $1$ to the middle value and then from the middle value back down to $1:$
$$\Omega = (1, 2, 3, \ldots, \frac{n+m}{2}-1, \frac{n+m}{2}, \frac{n+m}{2}, \frac{n+m}{2}-1,\ldots, 3, 2, 1).$$
Under the null hypothesis, $X$ and $Y$ are iid samples from a continuous distribution.  The Ansari-Bradley statistic is the sum of the ranks of the values from the first group.  Under the null hypothesis, its distribution is that of the sum of a simple random sample without replacement from the population of min ranks.
To apply the general observations we must center this population by subtracting its mean $(n+m+2)/4$ from each value.  The sum in (5) is a power sum evaluating to
$$\sum_{\omega\in\Omega}\omega^2 = 2\sum_{i=1}^{(n+m)/2} \left(i - \frac{n+m+2}{4}\right)^2 = \frac{1}{48}(n+m)(n+m - 2)(n+m + 2).$$
Consequently (5) tells us

$$\operatorname{Var}(\mathscr X) =  \frac{mn(n+m - 2)(n+m + 2)}{48(n+m-1)}.$$


As a quick check of every step in this analysis, we may repeatedly generate iid samples from any continuous distribution and compute the A-B test statistic.  The variance of all these simulated values ought to differ from the formula only by chance variation.  Here is an R implementation (using 10,000 simulated A-B statistics).
n <- 13
m <- 9        # Make sure n + m is even
n.sim <- 1e4  # Simulation size
AB <- apply(matrix(runif((n + m) * n.sim), ncol = n.sim), 2, 
            function(xy) ansari.test(xy[1:n], xy[(n+1):(n+m)])$statistic)
AB.var <- function(n, m) m * n * ((n + m)^2 - 4) / (48 * (n + m - 1))
signif(c(Theory = AB.var(n, m), Simulation = var(AB)), ceiling(log10(n.sim) / 2))

After executing set.seed(17) to initialize the random number generator, the output is

   Theory Simulation 
   56         56 


Experimenting with other values of $n$ and $m$ indicates the formula is correct.
