Determining if difference is significant I work on an ecommerce site, and I'm trying to determine a feature's impact on conversion.
This is not an AB test; rather, 100% of customers had access to the feature- whether or not they used the feature is indicated in "Segment" below.
In the past 12 months:




Segment
Converted
Didn't convert
Total




Used feature
629
20,570
21,097


Didn't use feature
243,273
8,131,398
8,374,671




Plain English: Of 21,097 customers who used feature, 629 or 2.98% converted. Of 8,374,671 who didn't use feature, 243,273 or 2.90% converted.
At first glance "Used feature" has higher conversion, but I'm wondering what other calculations I need to make a statistically sound conclusion.
New to stats.stackexchange so apologies if formatting is improper!

You can do the chi-squared test and get a p-value. (I haven't done the test, so I don't know what the p-value is.)
What you cannot do — without making a big assumption — is to claim that the feature increases the probability of conversion. It may be the case that users who were more likely to convert were also more likely to use the feature. In other words, the population of customers who use and don't use the feature may not the same.
This conundrum illustrates why A/B tests are useful: by randomizing customers to use the feature or not, we make sure that those who have access to the feature are comparable (exchangeable) with those who do not and so we can draw conclusions about the feature's effectiveness to increase conversion.
