What is the point of using a Bayesian prior? I do struggle with the most basic starting point of Bayesian statistics: why is using a prior useful?
It seems to me that if anything they hurt much more than help.
Moreover, Bayesians always say things like "the more evidence you get the less important a prior becomes". So why use them in the first place?
Especially if you have start with a prior that is really far-off, then you will be hurting your estimates.
To me a "Frequentist" approach seems much simpler and more straightforward.
I would like to discuss here an example that is very typical in introductory Bayesian course/explanations (e.g. this, this, or this)
Example 1 -- Are French people Rude?

Imagine that I am interested in estimating if French people are rude (in a binary way: Rude vs Non-Rude).
Imagine that the true parameter $\theta$ is 0.3, 30% of French are rude.
I have a random sample of 100 people with data on "rudeness".
A straightforward "Frequentist" approach would be: compute your confidence intervals on the sample and ... you done. We know what is the probability that the true $\theta$ lies within our CI, and we know that our sample average, on average, thanks to the Central Limit Theorem, will be close to the true parameter.
    true_theta = 0.3
    set.seed(111)

    # population
    X = rbinom(n = 10000, size = 1, prob = true_theta)

    # sample of 100
    x = sample(X, size = 100)
    p_hat = prop.table(table(x))[2]
    margin_error = 1.96 * sqrt( (p_hat*(1-p_hat)) / 100  
    ci_low = p_hat - margin_error
    ci_high = p_hat + margin_error
    data.frame(p_hat, ci_low, ci_high)

Now Bayesian will try to incorporate priors into this. Why?
We can imagine that most people will have beliefs that French are rude. (Using a Beta distribution) based on personal experience, someone would typically have a prior distribution like this: they know 10 French people and 7 are rude. So that is their prior.
The results are shown in the Figure above. On the bottom panel, I draw the Confidence Intervals around the sample mean.
It seems to me that the Frequentist approach will gives us a more precise and immediate answer to this question.
What am I missing here?
Example 2 -- Forecast Knock Out
Imagine I am interested in forecasting Knocked out (KO) in the Ultimate Fighting Championship (UFC).
My basic Frequentist approach would go like this.
Study the sport and see what variables play a role in predicting a fighter knocking another fighter (e.g. age of the fighters, winning streak, home advantage, …).
Then I will sample 30 UFC events and start build my model. I would use a simple logistic regression with maximum likelihood.
I don’t have prior per see, but I have a model of reality that I built on theoretical knowledge, like any scientists do, Bayesian or not.
My initial model would be that age and style of fighting (Muay Thai fighter vs BJJ) would be the most important factors predicting a KO.
I retrieve estimates from my logistic regression. Then I would cross-validate, take another sample and apply my initial model to the new data and see how it perform.
Let’s imagine that it doesn’t perform too well. I will then try to improve my model in studying more closely the sport. I find that accounting for an interaction between injury and age of the fighter is one of the most important predictors a KO. I re-run my regression with this and again cross validate, and now the model perform well.
Now I have a forecasting model telling me the probability of a KO and I can use it to bet money.
Furthermore, I know that certain special events, like a New Year's Eve event, will have more KOs, so I can adjust my model according to that fact.
Where would a prior, in the Bayesian sense, be useful here? Frequentists like all scientists use model to test reality and improve their models over time, but the difference is that they don’t put a formal initial probability on things. I still do not see why, you would want to do that.
    # code for the Figure
    theta_range <- seq(0, 1, by = 0.01)
    prior <- dbeta(x = theta_range, shape1 = 7, shape2 = 3)

    # observed success
    success_obs = table(x)[2]
    failure_obs = table(x)[1]

    # sampling distribution #
    likelihood <- dbinom(x = success_obs, size = 100, 
                           prob = theta_range) * 100

    # posterior 
    posterior_density <- likelihood * prior

   df <- data.frame(theta = theta_range,
                     likelihood = likelihood,
                     prior_dist = prior,
                     posterior_distribution = posterior_density)

    df_melt = melt(df, id.vars = 'theta')

    df_melt %>%
      ggplot(aes(x = theta, y = value, color = variable)) + 
      geom_line(size = 1.2) + 
      scale_x_continuous(breaks = seq(0, 1, by = 0.1)) + 
      geom_vline(xintercept = p_hat, size = 1.5) + 
      geom_vline(xintercept = ci_low, colour = 'gray', 
                                      size = 1.5) +
      geom_vline(xintercept = ci_high, colour = 'gray', 
                             size = 1.5) + theme_minimal() +
      ggtitle("Bayesian + Frequentist Confidence Intervals")


So an important area of work that I am involved in is decision making under uncertainty, particularly where money is placed at risk.  You are asking the wrong question.
The first question should be when and where should I use a Bayesian method and when should I not, maybe never, use a Bayesian method.  The subsidiary question would be about why you care about the answer.  An academic and a person solving an applied problem, even if the question is identical, probably should not use the same method.
Bayesian methods are good at helping me understand my beliefs and change them as new information comes in.  That me could be an us and the my could be an our.  If you have no beliefs on the topic, why do you even need to consider the topic?
Frequentist and Maximum Likelihood methods work really well when we lack prior belief because they minimize the maximum amount of risk that you will be facing.  Indeed, any two people that get the same result on a t-test with the same cutoff criterion should infer the same thing and decide the same thing.  That should not be true with Bayesian methods.
Now let us get back to my area of expertise.  How should I gamble money?
Well, it turns out that there are two very nice binding sets of rules that govern my behavior when money is involved.  The first is the Dutch Book Theorem.  If I were to massively oversimplify it, it says that if you gamble, you can never use a non-Bayesian method or it is possible to force you to take a sure loss.  In some cases, I can force you to take a sure loss 100% of the time.
If you are in finance, as I am, methods such as ordinary least squares can guarantee a loss to the user.  I do some interesting training exercises to show how that is done.  You get some seriously stunned looks when they lose and realized they could never have won.  Simple rule, you cannot use non-Bayesian methods when placing money at risk in a gambling or market type of situation.
The second rule comes because the Dutch Book Theorem and its converse imply that it is necessary to use a Bayesian method but it doesn't say that it is sufficient.  It turns out that it is not a sufficient criterion.
The second rule requires the use of a proper prior built from actual information.  The reason is that all admissible Frequentist rules are either the limiting form of a Bayesian rule or match the Bayesian outcome in every sample.  That leads to the question of whether or not the Frequentist solution inherits the Bayesian blessing when gambling, or if there are limits on Bayesian gambling.
The result is that there are limits on Bayesian gambling.  A market maker or bookie that is clueless about the game they are playing can be forced to take losses by an informed actor.  Unfortunately, I don't have citations where I am at. I print things out and put them in binders and I am not near my binders.  Nonetheless, if you did an academic search on the Dutch Book Theorem, you would likely come across them pretty quickly.
Your question is about the obvious problem of "what if the prior is wrong?"  Well, it can never be wrong because, for it to be valid, it should reflect your beliefs.  It isn't that the prior is wrong, it is that your beliefs are wrong.  The data should update your beliefs.  If you are very prejudiced, then they may be updated by an almost imperceptibly small amount.
Remember, in the Frequentist world, $\theta$ is a fixed point.  There is a correct single answer.  In the Bayesian world, $\theta$ is a random variable.  Even if there is a fixed, correct answer, it is quite likely that you will never find it.  That is okay, you have your distribution and if it is in the near neighborhood of reality, then all is well.
You cannot pass COVID on to others because COVID does not exist, therefore you do not need to isolate or quarantine if somebody tells you that you are exposed.
That is a belief.  It is wrong, but it is a belief.  Given the right type of data, you may interfere with or alter that belief.  Bayesian methods begin with you and where you are at.
How do you know that your sample of French people was representative?  Could you not have acquired a pleasant and welcoming subsample by chance?  Why trust the data alone?  Why not, instead, question if you are prejudiced?
