Biased coin game Assume, there's a 50% chance I get a fair coin and 50% I get a biased coin with 0.6 chance of getting heads.
Then, I get to toss the coin I got as many times as I want, but each toss costs a dollar.
The goal of the game is to determine if the coin is fair or not. Let's say I win 10000 dollars for the correct guess and lose 20000 dollars for the wrong one.
Again, I can toss the coin I got as many times as I want, but it's clear I don't want to toss it more than 10k times because then my profit is 0.
So, should I play the game? And what's the best number of tosses to maximize the profit/minimize loss?
What I've done so far, is found a Neyman-Pearson test statistic to see if the coin is biased:
$$T = \frac{n_0-n_1}{\sqrt{n_0+n_1}}$$
So, if I set the false alarm rate and missed signal rate I can find the number of tosses. But I'm not sure how to correctly set them.

I'd keep track of the Bayes factor for the two hypotheses over time, and make a decision once the evidence passes a threshold.
Let $F$ be the event that the coin is fair and $B$ be the event that the coin is biased. Let $n$ be the number of coin tosses observed, and let $h, t$ be the observed number of heads and tails. The Bayes factor here is
$$K(h, t) = \frac{P(F \mid h, t)}{P(B \mid h, t)} = \frac{\binom{n}{h}0.5^h0.5^t}{\binom{n}{h}0.6^h0.4^t} \frac{P(F)}{P(B)}.$$
You told us that there's a 50% chance we get a fair coin, so $P(F) = P(B) = 0.5$. The Bayes factor is a measure of how many times more likely one model is than another. My proposal is that you stop once $K(h, t)$ is greater than some threshold $T$, or less than $1/T$.
Cancelling out common terms and taking logarithms to make the Bayes factor nice to work with, we have
$$\log K(h, t) = n \log 0.5 - h \log 0.6 - t \log 0.4.$$
You stop once $\log K(h, t) > \log T$, or $\log K < -\log T$.
We need an estimate of how many observations we will see before we make a decision for a given threshold $T$. We can approximate this by taking the expectation of $\log K$. If $F$, then
$$\log K(h, t) = n \log 0.5 - (n/2) \log 0.6 - (n/2) \log 0.4 = n\log(0.5\cdot0.6^{-0.5}\cdot0.4^{-0.5}) \approx 0.02n.$$
For $\log K$ to exceed $\log T$, we will therefore need $0.02n \approx \log T$, or $n \approx 50 \log T$. I quickly checked that this is an OK approximation in R (code below). If we observe $\log K$ until it exceeds $1$ then we should wait approximately 50 steps. And running the code below shows that it is approximately 50:
set.seed(1)
samples = numeric(1000)
for (i in 1:1000) {
  samples[i] = min(which((1:1000)*log(0.5/0.4) - cumsum(rbinom(1000, 1, 0.5))*log(0.6/0.4) > 1))
}
mean(samples)
53.961

If $B$, then
$$\log K(h, t) = n \log 0.5 - 0.6n \log 0.6 - 0.4n \log 0.4 = n\log(0.5\cdot0.6^{-0.6}\cdot0.4^{-0.4}) \approx -0.02n.$$
For $\log K$ to be less than $-\log T$ we need $n \approx 50 \log T$ again.
If we stop when $K > T$, then
$$\frac{P(F \mid h, t)}{1 - P(F \mid h, t)} = T,$$
so
$$P(F \mid h, t) = \frac{T}{T + 1}.$$
Similarly, if we stop when $K < 1/T$ then
$$P(B \mid h, t) = \frac{T}{T + 1}.$$
Hence the probability that we make the right decision is $T/(T + 1)$.
The expected benefit to the game is the expected value of the \$10k vs -\$20k payoff given that the probability we're choosing correctly minus the expected number of turns until we get to threshold $T$ or $1/T$:
$$E_T = \frac{T}{T + 1}10{,}000 - \frac{1}{T + 1}20{,}000 - 50\log T.$$
Differentiation w.r.t. $T$ gives
$$\frac{1}{(T + 1)^2} 10{,}000 + \frac{1}{(T + 1)^2} 20{,}000 - \frac{1}{T} 50.$$
Setting this to $0$ and solving gives:
$$\hat{T} = 299 - 10\sqrt{894} \approx 600, \log \hat{T} \approx 6.4.$$
That's quite a high threshold! Here's a plot of the expected value depending on $T$ from Wolfram:

