How do you make sure the hypothesis test really works? Let’s suppose the following:
We are statistician, machine learner, data analyst, and so on, working for a company which produces tires for trucks. The company produces tires per hour with the following parameters:
$\mu = 80 $ tires per hour
$\sigma = 8$
Now the company introduced a new process, and they want to know whether this one really works or is a fraud. If the company adopts the new process, they have to invest thousands of dollars.
I make a hypothesis test to know if it’s worth the investment.
I draw a sample of 25 tires (with the new process), and I do the hypothesis test with significance level of  0.05. The mean with the new process is $\mu= 83$
$\alpha =0.05$
I do the hypothesis test:
$\frac{83-80}{\frac{8}{\sqrt{25}}}= 1.875$
Acording to normal table, our p-value is 0.031. So we have approx. 3.1% probability to make type I error. Therefore, we say to the company that they should invest thousands of dollars in the new technique.
My question is: what happens if we had bad luck and by chance we take the most extreme sample, and in reality the new technique to produce tires doesn't work?
In the real life what would you do?
Would you take more random samples and the media of all those samples?
In case you have just one opportunity and you can't take more samples: do you lower the significance level to 0.01?
What other statistics techniques or machine learning techniques would you use?


My question is: what happens if we had bad luck and by chance we take
the most extreme sample, and in reality the new technique to produce
tires doesn't work?

It can happen, you can be unlucky. The bad news is that not much can be done about verifying it. To be more precise, you can collect more data to minimize the chance of bad luck. However, if you could collect more data, why not do this in the first place?
The good news is that hypothesis tests are designed for handling such cases by themselves. This is why before the experiment, you set the $\alpha$ and $\beta$ levels (see, e.g. Banerjee et al, 2009), so to decide what are the acceptable risks for the type I (false positive) and type II (false negative) errors (see type-i-and-ii-errors). Before the experiment, you need to conduct power analysis (see power-analysis) and decide on the appropriate sample size. As you can see, we are back at collecting enough data. In each case, there still is a risk that you get incorrect results, but this is how you bound the risk to acceptable levels.
