The Two-proportion Z-test (or, Two-sample proportion Z-test) is a statistical method used to determine whether the difference between the proportions of two groups, coming from a binomial distribution is statistically significant. This approach relies on the assumption that the sample proportions follow a normal distribution under the Central Limit Theorem, allowing the construction of a z-test for hypothesis testing and confidence interval estimation. It is used in various fields to compare success rates, response rates, or other proportions across different groups.
Hypothesis test.
The z-test for comparing two proportions is a Statistical hypothesis test for evaluating whether the proportion of a certain characteristic differs significantly between two independent samples. This test leverages the property that the sample proportions (which is the average of observations coming from a Bernoulli distribution) are asymptotically normal under the Central Limit Theorem, enabling the construction of a z-test.
The test involves two competing hypotheses:
The z-statistic for comparing two proportions is computed using:
formula_5
Where:
The pooled proportion is used to estimate the shared probability of success under the null hypothesis, and the standard error accounts for variability across the two samples.
The z-test determines statistical significance by comparing the calculated z-statistic to a critical value. E.g., for a significance level of formula_14 we reject the null hypothesis if formula_15 (for a two-tailed test). Or, alternatively, by computing the p-value and rejecting the null hypothesis if formula_16.
Confidence interval.
The confidence interval for the difference between two proportions, based on the definitions above, is:
formula_17
Where:
This interval provides a range of plausible values for the true difference between population proportions.
Using the z-test confidence intervals for hypothesis testing would give the same results as the chi-squared test for a two-by-two contingency table. Fisher’s exact test is more suitable for when the sample sizes are small.
Notice how the variance estimation is different between the hypothesis testing and the confidence intervals. The first uses a pooled variance (based on the null hypothesis), while the second has to estimate the variance using each sample separately (so as to allow for the confidence interval to accommodate a range of differences in proportions). This difference may lead to slightly different results if using the confidence interval as an alternative to the hypothesis testing method.
Minimum detectable effect (MDE).
The minimum detectable effect (MDE) is the smallest difference between two proportions (formula_19 and formula_20) that a statistical test can detect for a chosen Type I error level (formula_21), statistical power (formula_22), and sample sizes (formula_8 and formula_9). It is commonly used in study design to determine whether the sample sizes allows for a test with sufficient sensitivity to detect meaningful differences.
The MDE for when using the (two-sided) z-test formula for comparing two proportions, incorporating critical values for formula_21 and formula_22, and the standard errors of the proportions:
formula_27
Where:
The MDE depends on the sample sizes, baseline proportions (formula_31), and test parameters. When the baseline proportions are not known, they need to be assumed or roughly estimated from a small study. Larger samples or smaller power requirements leads to a smaller MDE, making the test more sensitive to smaller differences. Researchers may use the MDE to assess the feasibility of detecting meaningful differences before conducting a study.
The Minimal Detectable Effect (MDE) is the smallest difference, denoted as formula_32, that satisfies two essential criteria in hypothesis testing:
Given that the distribution is normal under the null and the alternative hypothesis, for the two criteria to happen, it is required that the distance of formula_37 will be such that the critical value for rejecting the null (formula_38) is exactly in the location in which the probability of exceeding this value, under the null, is (formula_21), and also that the probability of exceeding this value, under the alternative, is formula_35.
The first criterion establishes the critical value required to reject the null hypothesis. The second criterion specifies how far the alternative distribution must be from formula_38 to ensure that the probability of exceeding it under the alternative hypothesis is at least formula_35.
Condition 1: Rejecting formula_43
Under the null hypothesis, the test statistic is based on the pooled standard error (formula_44):
formula_45
formula_46 might be estimated (as described above).
To reject formula_43, the observed difference must exceed the critical threshold (formula_48) after properly inflating it to the SE:
formula_49
If the MDE is defined solely as formula_50, the statistical power would be only 50% because the alternative distribution is symmetric about the threshold. To achieve a higher power level, an additional component is required in the MDE calculation.
Condition 2: Achieving power formula_35
Under the alternative hypothesis, the standard error is (formula_52).
It means that if the alternative distribution was centered around some value (e.g., formula_38), then the minimal formula_37 must be at least larger than formula_55 to ensure that the probability of detecting the difference under the alternative hypothesis is at least formula_35.
Combining conditions
To meet both conditions, the total detectable difference incorporates components from both the null and alternative distributions. The MDE is defined as:
formula_57
By summing the critical thresholds from the null and adding to it the relevant quantile from the alternative distributions, the MDE ensures the test satisfies the dual requirements of rejecting formula_43 at significance level formula_21 and achieving statistical power of at least formula_35.
Assumptions and conditions.
To ensure valid results, the following assumptions must be met:
The z-test is most reliable when sample sizes are large, and all assumptions are satisfied.
Software implementation.
R.
Use codice_1 with continuity correction disabled:
prop.test(x = c(120, 150), n = c(1000, 1000), correct = FALSE)
Output includes z-test equivalent results: chi-squared statistic, p-value, and confidence interval:
 2-sample test for equality of proportions without continuity correction
data: c(120, 150) out of c(1000, 1000)
X-squared = 3.8536, df = 1, p-value = 0.04964
alternative hypothesis: two.sided
95 percent confidence interval:
 -5.992397e-02 -7.602882e-05
sample estimates:
prop 1 prop 2 
 0.12 0.15 
Python.
Use codice_2 from statsmodels:
from statsmodels.stats.proportion import proportions_ztest
z, p = proportions_ztest([120, 150], [1000, 1000], 0)