How to test if two events with unknown probabilities are different or not Let's say I have an "unfair die" with hundreds of possible results, and each result may have a different probability. However, I am interested only in two of these results and I want to test if they have the same probability or not.
Real data: After (at least) 500 rolls, the result A appeared 10 times and the result B appeared 44 times. How should I test this? And how should I involve the sample size? (I did at least 500 rolls, but they might even be 2000).
I've read about testing the fairness of a die using chi-square, but I don't know how to handle there the expected values, especially in relation to the number of times where the result was neither A or B.

A Bayesian approach to the problem works out neatly. The reference posterior (see equation 6) of $\phi\equiv{p_A/p_B}$ is
$\phi^{ref}\sim{}B'(\phi;x_A+1/2,x_B+1/2)$,
where $B'$ is the beta prime distribution and $x_A$ and $x_B$ are the number of observations of $A$ and $B$ from the independent trials. Notice that the posterior distribution of $\phi$ is parameterized only by the counts of observations of $A$ and $B$, so the results of the analysis are the same whether the counts came from 500 rolls or 2000 rolls.
The beta prime CDF is the regularized incomplete beta function,
$I_{\frac{\phi}{\phi+1}}(x_A+1/2,x_B+1/2)$
The posterior likelihood that $\phi>1$ (i.e., $p_A>p_B$) is
$1-I_{1/2}(10.5,44.5)\approx{7.97\times10^{-7}}$
In R:
pbeta(0.5, 10.5, 44.5, lower.tail = FALSE)
#> [1] 7.972998e-07


As Bernardo and Ramón point out, the analysis works out similarly with the parameter $\theta_{A|AB}\equiv\frac{\phi}{\phi+1}$, which is the probability that the die shows $A$ given that the result of the roll is either $A$ or $B$. The reference/Jeffreys posterior of the probability of a Bernoulli trial is given by the beta distribution:
$Be(\theta_{A|AB};x_A+1/2,x_B+1/2)$
