What Does a P-Value Really Mean?

P-values are among the most commonly reported—and most commonly misunderstood—numbers in medical and scientific research.

You may have seen statements such as:

“The result was statistically significant because the p-value was less than 0.05.”

But what does that actually mean?

Does it mean the treatment works? Does it mean there is only a 5% chance that the result is wrong? Does it mean the finding is clinically important?

Not necessarily.

What Is a P-Value?

A p-value measures how compatible the observed data are with a specific statistical assumption, usually the null hypothesis.

The null hypothesis often states that there is no difference between two groups, no treatment effect, or no association between two variables.

In simple terms, a p-value asks:

If there were truly no difference or association, how unusual would results like these be?

A small p-value suggests that the observed results would be relatively unusual under the null hypothesis.

For example, a p-value of 0.03 means that, assuming the null hypothesis and the statistical model are correct, there is approximately a 3% probability of observing results this extreme or more extreme.

It does not mean that there is a 3% probability that the null hypothesis is true.

That distinction is extremely important.

Why Is 0.05 Commonly Used?

Researchers often compare the p-value with a predefined significance level, commonly called alpha.

The most frequently used significance level is 0.05.

When:

• p < 0.05, the result is often described as statistically significant.

• p ≥ 0.05, the result is often described as not statistically significant.

However, 0.05 is a convention, not a magical boundary separating truth from falsehood.

A result with a p-value of 0.049 is not fundamentally different from a result with a p-value of 0.051. Treating one as a definite success and the other as a complete failure can be misleading.

The p-value should be interpreted as part of the overall evidence, not as a simple pass-or-fail test.

Why Are P-Values Important?

Despite their limitations, p-values remain important in medical research, clinical trials, public policy, and regulatory submissions.

They provide a standardized method for evaluating whether an observed finding may be inconsistent with the null hypothesis.

In a clinical trial, for example, a p-value may help assess whether the difference between a treatment and a control group is likely to be more than random variation.

P-values are also commonly included in:

• Clinical trial reports

• Regulatory submissions

• Health technology assessments

• Medical publications

• Public health evaluations

• Policy and funding decisions

Regulatory and policy decisions generally should not depend on a p-value alone. They may also consider the size of the treatment effect, confidence intervals, safety findings, study quality, clinical relevance, and the totality of available evidence.

Nevertheless, p-values remain part of many established statistical and regulatory decision-making frameworks. Therefore, even those of us who prefer other measures still need to understand and use them correctly.

Common Misinterpretations of P-Values

1. “The p-value is the probability that the null hypothesis is true.”

This is incorrect.

A p-value is calculated while assuming that the null hypothesis is true. It does not calculate the probability that the hypothesis itself is true or false.

2. “A statistically significant result must be clinically important.”

Statistical significance and clinical importance are not the same.

A very small treatment difference may produce a small p-value in a large study, even when the difference has little practical importance.

For example, a treatment might reduce a symptom score by only 0.2 points but still produce a statistically significant result because thousands of participants were included.

The result may be statistically detectable without being meaningful to patients.

3. “A non-significant result proves that there is no effect.”

A non-significant result does not prove that the groups are identical.

It may mean that:

• The true effect is small.

• The study did not include enough participants.

• The data were highly variable.

• The measurement was imprecise.

• The study design was not sensitive enough to detect the effect.

“Not statistically significant” should not automatically be translated into “no difference.”

4. “A smaller p-value means a larger effect.”

Not necessarily.

The p-value is influenced by both the size of the effect and the size of the study.

A tiny effect in a very large study may have a very small p-value. A clinically meaningful effect in a small study may have a larger p-value because the estimate is less precise.

To understand the importance of a result, we need to examine the estimated effect itself.

5. “P < 0.05 means there is a 95% chance the result is correct.”

This is also incorrect.

The p-value does not directly provide the probability that the study conclusion is correct. That would require additional assumptions and a different statistical framework.

6. “Testing many outcomes does not affect the results.”

When researchers perform many statistical tests, the probability of obtaining at least one statistically significant result purely by chance increases.

For example, if 20 unrelated outcomes are tested using a significance level of 0.05, it would not be surprising to see one significant result simply due to random variation.

This is why clinical trials should clearly define their primary and secondary outcomes in advance and use appropriate methods to address multiple testing.

Why I Prefer Confidence Intervals

Personally, I prefer confidence intervals because they usually provide more useful information than a p-value alone.

A p-value mainly helps answer whether the data appear inconsistent with a particular null hypothesis.

A confidence interval helps show:

• The estimated size of the effect

• The direction of the effect

• The precision of the estimate

• The range of values reasonably compatible with the data

Suppose a study reports that a treatment reduced blood pressure by 4 mmHg, with a 95% confidence interval from 1 mmHg to 7 mmHg.

This tells us much more than simply reporting p = 0.02.

We can see that the estimated reduction is 4 mmHg. We can also see the uncertainty around that estimate and consider whether the possible effects would be clinically meaningful.

Now consider another study reporting a reduction of 4 mmHg with a 95% confidence interval from −3 mmHg to 11 mmHg.

The estimated effect is the same, but the second study is much less precise. The interval includes the possibility of no benefit, a small harmful effect, and a meaningful benefit.

The confidence interval provides a clearer picture of what the data do and do not tell us.

P-Values and Confidence Intervals Should Work Together

The goal should not necessarily be to eliminate p-values.

Instead, p-values should be reported and interpreted alongside:

• Effect estimates

• Confidence intervals

• Sample sizes

• Clinical relevance

• Study design

• Data quality

• Safety outcomes

• Prior evidence

For example, rather than writing:

The treatment produced a statistically significant improvement, p = 0.03.

A more informative statement would be:

The treatment improved the outcome by an estimated 4.2 points compared with the control, with a 95% confidence interval from 0.5 to 7.9 points and a p-value of 0.03.

The second statement tells the reader the estimated size, direction, uncertainty, and statistical evidence.

The Bottom Line

A p-value is not the probability that a result is true, false, or caused by chance.

It is a measure of how unusual the observed data would be under a specified null hypothesis and statistical model.

P-values can be useful, particularly in clinical trials, regulatory work, policy evaluation, and formal statistical decision-making. However, they should never be interpreted alone.

A statistically significant result may not be clinically important, and a non-significant result does not prove that there is no effect.

My preference is to focus more heavily on effect estimates and confidence intervals because they show both the magnitude of the finding and the uncertainty surrounding it.

But whether we like p-values or not, they remain an important part of medical research. The key is not simply to calculate them—it is to interpret them correctly.