Fisher's Exact Test
Fisher's Exact Test is a classical statistical estimate of the significance of association between two classifications of categorical data—this is the two-sided test.*
Enter numbers in the boxes to the right corresponding to counts for each category for two groups and press "Compute" to see a corresponding p value that estimates how likely the association is— the smaller the p value, the more likely there is an association between the groups and categories.
For example, let's say you want to test if there is an association between whether or not email is opened and email subject line version. Send out your emails to sufficiently randomized recipients at approximately the same times to avoid confounding time effects. Such a test might look like this:
This version of Fisher's exact test is limited to two groups. Generally speaking, logisitc regression is a much more flexible approach that yields similar results. Logistic regression can test many groups as well as effects from additional variables.
*(It tests a null hypothesis of independence of rows and columns in a contingency table with fixed marginals.)
Fisher's Exact Test is a classical statistical estimate of the significance of association between two classifications of categorical data—this is the two-sided test.*
Enter numbers in the boxes to the right corresponding to counts for each category for two groups and press "Compute" to see a corresponding p value that estimates how likely the association is— the smaller the p value, the more likely there is an association between the groups and categories.
For example, let's say you want to test if there is an association between whether or not email is opened and email subject line version. Send out your emails to sufficiently randomized recipients at approximately the same times to avoid confounding time effects. Such a test might look like this:
The estimated p value for this example, 0.0882, is generally considered not very significant leading to a conclusion that there is not a strong association between open rate and the choice of these two email subject lines.
*(It tests a null hypothesis of independence of rows and columns in a contingency table with fixed marginals.)
| Category A | Category B | |
|---|---|---|
| Group 1 | ||
| Group 2 | ||
P values estimate the chance you're wrong
Let's say you get a p value of 0.01. You conclude that there is a significant association between groups and categories. Yay!
That p value of 0.01 means that there is an estimated 1-out-of-100 chance that your conclusion is wrong (under the assumptions baked in to this test). That's reasonably unlikely.
Note p values—and significance testing generally—can be misleading. You need to think about your test carefully, make sure randomization is done as well as possible and potential confounding factors you know of are limited. And ideally you should replicate your test without changing the set-up a few times to make sure you're not fooling yourself. Try to avoid the "I've got statistical significance and I'm outta here" attitude coined by Andrew Gelman and Deborah Mayo in Confirmationist and falsificationist paradigms of science.
Let's say you get a p value of 0.01. You conclude that there is a significant association between groups and categories. Yay!
That p value of 0.01 means that there is an estimated 1-out-of-100 chance that your conclusion is wrong (under the assumptions baked in to this test). That's reasonably unlikely.
Note p values—and significance testing generally—can be misleading. You need to think about your test carefully, make sure randomization is done as well as possible and potential confounding factors you know of are limited. And ideally you should replicate your test without changing the set-up a few times to make sure you're not fooling yourself. Try to avoid the "I've got statistical significance and I'm outta here" attitude coined by Andrew Gelman and Deborah Mayo in Confirmationist and falsificationist paradigms of science.