Start with your constraint
A deadline plus daily eligible traffic determines the audience available. A fixed audience determines the detectable effect. A target effect determines required audience, and traffic translates it into enrollment time. All three entry points use the same calculation.
Effects and business relevance
A 10% relative uplift from a 10% baseline means a new rate of 11%: a gain of one percentage point. The minimum detectable effect (MDE) is the true effect associated with the chosen power. It is not a guarantee of significance or a cutoff below which effects are impossible to detect.
Alpha, beta and test direction
Alpha controls false rejections under the null. Beta is the missed-effect probability at a specified true effect; power is one minus beta. A one sided test asks a predeclared directional question. A two sided test detects either direction and splits total alpha across both tails. Choose the primary metric, direction, alpha and stopping rule before examining results.
Historical effect rate and prevalence
Your historical effect rate is tests with a detected effect divided by all past tests. The program model uses this rate as an explicit proxy for the underlying prevalence of real effects. It is useful for a scenario, but significant outcomes alone do not establish ground truth.
Expected false positives are upcoming test count × (1 − assumed prevalence) × current effective alpha. This is a program-level scenario, not a posterior probability for an individual result.
Planning method
Normal approximation for two independent proportions: pooled variance under the null and separate variances under the specified alternative. A numerical solver finds the required audience or effect. Sample counts round upward. Sparse expected outcomes are flagged because this approximation can be inaccurate.
Results and uncertainty
The results view uses an uncorrected score test. Confidence limits come from inversion of that score test using constrained maximum likelihood for the difference in rates. A one sided test shows the matching confidence bound; a two sided test shows an interval. Sparse observed counts suppress approximate inference. The allocation check uses a chi-square approximation with one degree of freedom and flags p < 0.001 for investigation.
A p value is not the probability B is better. An inconclusive result does not establish no effect. Commercial relevance and statistical evidence are separate. A significant one sided improvement does not prove the effect exceeds a business threshold.
Timing and repeated tests
Traffic per selected period must be eligible independent participants. For planning, a month is 30 days, a quarter is 90 days and a year is 365 days; traffic is converted to an average daily rate. Repeated visits or repeated emails to the same person are not extra independent observations. A single fixed campaign does not gain sample size by waiting longer. Outcome maturation is separate from enrollment. This tool assumes a fixed stopping point; unplanned significance-based early stopping needs a sequential method.
Multiple hypotheses
If several hypotheses support one decision, the optional Bonferroni setting divides the family alpha budget by the number of hypotheses. It changes sample planning and result thresholds. Counting unrelated experiments over a year does not automatically define such a family.
Privacy and hosting
Calculations run in your browser. There are no accounts, tracking scripts, or data uploads. Exports are generated on your device. The portable site consists of static files and can run on your existing hosting.