How do you make sure the hypothesis test really works? Let’s suppose the following:
We are statistician, machine learner, data analyst, and so on, working for a company which produces tires for trucks. The company produces tires per hour with the following parameters:
$\mu = 80 $ tires per hour
$\sigma = 8$
Now the company introduced a new process, and they want to know whether this one really works or is a fraud. If the company adopts the new process, they have to invest thousands of dollars.
I make a hypothesis test to know if it’s worth the investment.
I draw a sample of 25 tires (with the new process), and I do the hypothesis test with significance level of  0.05. The mean with the new process is $\mu= 83$
$\alpha =0.05$
I do the hypothesis test:
$\frac{83-80}{\frac{8}{\sqrt{25}}}= 1.875$
Acording to normal table, our p-value is 0.031. So we have approx. 3.1% probability to make type I error. Therefore, we say to the company that they should invest thousands of dollars in the new technique.
My question is: what happens if we had bad luck and by chance we take the most extreme sample, and in reality the new technique to produce tires doesn't work?
In the real life what would you do?
Would you take more random samples and the media of all those samples?
In case you have just one opportunity and you can't take more samples: do you lower the significance level to 0.01?
What other statistics techniques or machine learning techniques would you use?

It may help to further define "the new process works" and "fraud". Maybe 83 tires/hr is not very different from 80 tires/hr. Maybe its only worth it if the difference is 15 or something. Similarly, I would say the process is a "fraud" if the rate was much lower than 80. You can setup an equivalence bound and do two one-sided test for equivalence to establish equivalence of the two processes.
You can also look into utility theory rather than null hypothesis testing. You can answer questions like  how much do we need to make the rate increase by to ensure we recoup the loss in investment after X days.
