AB test not significant in statistical testing, but significant due to sample size? I'm running A/B testing on YouTube thumbnails using proprietary software. After only two days (one day using one thumbnail and another day using the other), my results come back as favoring the new thumbnail with something upward of a 96% confidence level.
However, it's only been two days and the sample sizes are what I would say are small. It seemed odd to me so I ran it really quickly and my p-value was 0.0706 which shows it wasn't significant.
Data for A
Visitors: 24
Conversions: 6
Data for B
Visitors: 29
Conversions: 2
I asked what is causing the discrepancies between previous calculations when I was skeptical and received an answer from a user (the company has been non-responsive to my ticket)

TubeBuddy's A/B testing simply alternates the thumbnails each day
until it reaches statistical significance (if that's the option you
selected). You don't really need a two-sampled test because the sample
size itself determines whether it's significant.
But if you're curious as to whether the sampling that made up the two
groups caused the results to be biased, you could also ways run the
test again to see if you get the same outcome.

I'm still having a hard time understanding how it's significant and what he means about the sample size making it significant when I have such a small sample size. It seems counterintuitive. What am I not getting? (Explain it like I'm five?)

I think you are doing a questionable statistical test here. What happens on a particular day might be related. E.g. day 1 is a Sunday, so people have a lot of time and click on stuff and convert, day 2 in a Monday and people are more busy and don't. This does not mean that what you did on day 1 is better than what you did on day 2. A sensible analysis here should account for day-to-day variability (i.e. a simple comparison of proportion is probably inappropriate). E.g. a random effects logistic regression with a random day effect on the intercept could be an option.
Randomly assigning one strategy for some users and another for others (on the same day) tends to be better. Alternatively, if you have enough days that you are willing to wait for, randomize days (perhaps in blocks to not have excessively long runs with one approach). Alternating assignment might be okay, but can in some circumstances have problems that randomization doesn't have.
Finally, the company's approach of keeping on going until you have significance invalidates the hypothesis test and if you only go long enough you will always reach significance with this approach even if there's in truth absolutely no difference.
PS: I don't know what you mean by "...upward of 96% significant."
PPS: I also have no idea what the company is talking about with "You don't really need a two-sampled test because the sample size itself determines whether it's significant." Perhaps there's some meaningful thing at the bottom of what they are trying to say, but it does not really make any sense as written.
