Statistical power complement:
formula and worked example

The chance of seeing a real effect

Statistical power is the probability that a test will reject the null hypothesis when a real effect of a specified size exists. Beta is the opposite: the probability of a Type II error, failing to detect an effect that is there. Power is simply 1 minus beta, shown here as a percentage.

Power is something you plan, not something you discover. It depends on the sample size, the size of the effect you care about, the variability of the data and the significance level you will use. Chosen before data collection, it tells you whether the study is capable of answering its own question.

A study with low power can easily miss a real effect and, worse, when it does find one the estimate tends to be exaggerated. That is why funders and ethics boards ask for a power justification.

It helps to see where beta sits among the four possible outcomes of a test. If an effect exists and the test detects it, that is power. If the effect exists and the test misses it, that is beta. If no effect exists, the test either correctly stays quiet or raises a false alarm with probability alpha. Only the first two depend on there being a real effect.

The formula and its single input

• Beta: the probability of a Type II error, the chance of missing a true effect of the size you specified, expressed as a proportion.

The conversion is trivial; the difficulty is in obtaining a sound beta. That comes from a power analysis, which links beta to the sample size per group, the expected effect size, the standard deviation (or baseline rate) and alpha, the Type I error rate.

Worked example: beta of 0.2

A team plans a two-group experiment. Their power analysis, assuming the smallest effect they care about, gives a Type II error rate of 0.2.

1. 1Take beta from the planning calculation: 0.2.
2. 2Subtract from one: 1 - 0.2 = 0.8.
3. 3Multiply by 100: 80%.
4. 4Interpret: if the assumed effect is real, the study would detect it about 8 times in 10.

Why 80% appears so often

Eighty per cent power is widely used as a minimum target, along with a 5% alpha, which implies a tolerance for a Type II error four times that for Type I. The figure grew out of a convention that false alarms are worse than misses, and it is a compromise between precision and cost, not a mathematical requirement. Some fields ask for 90%, particularly when a missed effect would be costly.

Reading 80% correctly means keeping the conditions in mind. It is the power for one particular effect size. If the true effect is smaller than assumed, the real power is lower, and if it is larger, the power is higher.

It also means that even a well-designed study will miss a real effect one time in five. A single non-significant result from an 80%-powered study is therefore weak evidence that no effect exists.

Planning teams sometimes work the other way round: fix power at 80%, fix alpha, and solve for the sample size or for the smallest effect the study can reliably detect. That minimum detectable effect is a useful number to quote in a protocol, because it tells readers which effects the study was never able to see.

What a slightly higher beta costs

A beta of 0.22 gives 78% power, two points lower. Because the relationship is a straight subtraction, every point of Type II error is a point of lost power. The practical path to lower beta is to increase the sample size, use a more precise measurement, or accept a larger minimum detectable effect. Sample size has diminishing returns: gaining the last few points of power typically requires a disproportionately larger study.

How power is misused

• Computing power after the study from the observed effect. This 'observed power' is a function of the p-value and tells you nothing new.
• Treating 80% as a guarantee of success, rather than a probability conditional on the effect really being the size assumed.
• Confusing beta with alpha, or assuming a 5% significance level means 95% power. The two are separate quantities.
• Ignoring attrition. If 20% of participants drop out, the effective sample size and the real power fall below the planned figure.
• Planning for a large effect because it gives a small, affordable sample, when the realistic effect is much smaller.

Using power in planning

Imagine comparing a checkout page change expected to lift conversion by a small amount. If the planned sample gives only 50% power, there is a coin-flip chance of missing a genuine improvement, and teams may wrongly abandon a good idea. Raising power to 80% by running the test longer is usually cheaper than the cost of discarding a real gain.

The same reasoning guards against over-reading a quiet result. A study that reports 'no significant difference' with 40% power has said very little, while the same words from a 95%-powered study mean something real. Always ask for the power, or for the confidence interval, before accepting a negative finding.