Which metric(s) can I use to evaluate how well a group of binary models agree in their predictions? My request is best explained with an example. Suppose the upcoming week of (American) football matches are
Bills vs Jets
Saints vs Falcons
Rams vs Lions
Packers vs Bears
Chiefs vs Chargers
Giants vs Cowboys

I ask three guys and three girls "who will win?" Their responses are
guy 1: Bills and Rams
guy 2: Jets, Rams, and Giants
guy 3: Rams, Chiefs and Cowboys
girl 1: Jets, Rams, and Giatns
girl 2: Jets, Rams, and Bears
girl 3: Packers and Chiefs
I'm looking for a metric that can help me answer the following:

*

*How well do the guys agree?

*How well do the girls agree?

*Which gender has a higher level of agreement?

Notes

*

*I do not know wins each game (nor do I care).

*Each person doesn't predict on the exact same set of games


Current Approach
My current idea is to implement the following algorithm.
Let $P_i$ be the set of predictions for model $i$. Given $P_1$, $P_2$, ..., $P_N$,
For $i$ from 0 to N:
Assume $P_i$ is the "source of truth". Now I can calculate $F_i$ as the average F score amongst the other predictors $\{P_k\}_{k \neq i}$. This tells me how well the other predictors agree with $P_i$.
Finally, I just can just average the F scores (perhaps weighted by sample size).
I suspect there's a more elegant approach to this..

For convenience, I'll use a simpler version of your example. Consider the teams $A,B,C,$ and $D$. There are ${4 \choose 2} = 6$ possible matches between these teams:




Match Number ($M$)
Team 1 ($T = 0$)
Team 2 ($T = 1$)




1
$A$
$B$


2
$A$
$C$


3
$A$
$D$


4
$B$
$C$


5
$B$
$D$


6
$C$
$D$




Let $M$ correspond to the match number, such that $M = 1$ corresponds to the match $A$ vs. $B$, for example. Let $G$ denote the gender of the person being asked, such that $G = 0$ corresponds to boy and $G = 1$ corresponds to girl. Also, let $N$ denote the number assigned to the gender, such that $G = 0$ and $N = 3$ corresponds to "boy 3", for example. Finally, let $T$ represent the prediction of the team that wins, such that $T = 0$ corresponds to team 1 and $T = 1$ corresponds to team 2.
As an example, the probability $P(T = 1 \mid M = 4, G = 1, N = 2)$ corresponds to the probability that girl 2 predicted that team $C$ will win in the match $B$ vs. $C$. Note that, regardless of the values of $M,G,$ and $N$, $T$ is a Bernoulli random variable.
The Phi coefficient is just the Pearson's correlation coefficient for two Bernoulli random variables. In other words, it is the normalized covariance of two Bernoulli random variables. In our case, we are interested in the normalized covariances for two triplets of Bernoulli random variables: $(T \mid G = 0,N = 1,T \mid G = 0,N = 2,T \mid G = 0,N = 3)$ and $(T \mid G = 1,N = 1,T \mid G = 1,N = 2,T \mid G = 1,N = 3)$.
As for how to compute these two normalized covariances, you can get started here.
