Does every hypothesis testing require statistical analysis? Assuming I am conducting an AB testing on a website to see if there is an increase on user click thru rate. I know there are many different kinds of tests such as t-test, ANOVA, etc. Is it required for me to use these tests if my result is obvious that there is a substantial difference? In other words, when should I use statistical test during my AB testing?

This quote answers your question:

Rejection of a true null hypothesis at the 0.05 level will occur only one in 20 times. The overwhelming majority of these false rejections will be based on test statistics close to the borderline value. If the null hypothesis is false, the inter-ocular traumatic test ["hit between the eyes"] will often suffice to reject it; calculation will serve only to verify clear intuition.

W. Edwards, Harold Lindman, Leonard J. Savage (1962) Bayesian Statistical Inference for Psychological Research. University of Michigan. Institute of Science and Technology. (from here)
