The p-value meaning, in one sentence: it is the probability of seeing data at least as extreme as yours if the null hypothesis of no real difference were true. So p = 0.03 says such data would turn up about 3% of the time under no effect. It is not the chance the null is true, and it says nothing about whether the effect is big enough to act on.
Why the p-value meaning matters on the CBDA exam
The IIBA Certification in Business Data Analytics (CBDA) exam tests interpretation more than calculation. A wrong p-value reading in Analyze Data (16% of questions) becomes a wrong recommendation in Interpret and Report Results and Use Results to Influence Business Decision Making (20% each).
| CBDA fact | Value |
|---|---|
| Questions | 75, multiple-choice, scenario-based |
| Time | 120 minutes |
| Pass mark | Not published; pass or fail shown on screen |
| Analyze Data | 16% of questions |
| Interpret and Report Results | 20% of questions |
Last verified: October 2026 on the IIBA CBDA page and the CBDA Certification Handbook (May 2026).
How the p-value works
The IIBA's public Introduction to Business Data Analytics: A Practitioner View (2019, §2.3.4) says that analyzing data "requires extensive application of statistics" but does not teach p-values itself. For the definition, go to statistical references.
The NIST/SEMATECH e-Handbook of Statistical Methods defines the p-value as the probability of a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. It recommends fixing the threshold, the significance level alpha, before running the test. With alpha = 0.05, you reject the null when p falls below 0.05.
The American Statistical Association's 2016 statement on p-values lists six principles. Two matter most for exam questions:
- P-values "do not measure the probability that the studied hypothesis is true."
- A p-value "does not measure the size of an effect or the importance of a result."
The p-value is a statement about the data given a hypothesis. Most wrong answers flip it into a statement about the hypothesis given the data.
Worked example: the checkout A/B test
An e-commerce team tests a new checkout flow against the current one. Alpha was set at 0.05 before launch. After two weeks, conversion is 3.10% on the old flow and 3.18% on the new one, with p = 0.03.
What the analyst can say:
- The result is significant at the 5% level. If the flows truly converted the same, a gap this large or larger would appear in about 3% of repeated tests.
- The observed lift is 0.08 percentage points. That effect size needs its own judgment: its revenue value and its confidence interval.
- The decision depends on business value. A small reliable lift may be worth shipping if the flow is cheap to maintain.
What the analyst cannot say: "there is a 97% chance the new flow is better." The p-value does not give that number.
One more check: was the test stopped early? Evan Miller's 2010 essay How Not To Run an A/B Test shows that stopping at the first significant peek inflates false positives: "Repeated significance testing always increases the rate of false positives."
Why each wrong option fails
The original question asks what the analyst can conclude from p = 0.03 at a 0.05 significance level.
| Option | Why it is tempting | Why it fails |
|---|---|---|
| A. 3% chance the null is true | Sounds like a direct reading | Inverts the logic; p assumes the null, it does not rate it |
| B. 97% likely the new flow is better | Feels like the complement of A | Same inversion, plus a claim about direction and size |
| C. Data this extreme would occur about 3% of the time with no real difference | Sounds technical | Correct: a statement about data under the null |
| D. The improvement is big enough to act on | Significance sounds like importance | Confuses statistical significance with effect size |
A and B sound like plain-English versions of C, but they are claims about hypotheses; C is a claim about data.
What practitioners say
Spotify's Confidence glossary entry on p-values states that a p-value "does not tell you the probability that the change worked." It also warns that reporting the best of many metrics inflates false positives.
The Statsig team's 2025 guide makes the D point: with enough traffic, tiny effects become significant, so check effect size before shipping. In their words, significance "indicates a low chance result, not necessarily a meaningful difference."
Mufaddal Haidermota, writing about the CBDA on Medium in 2023, describes 75 scenario-based questions and notes that the certification does not go deep into data science or mathematical models. You are judged on what you conclude, not on computation.
Common misconceptions
- "p = 0.03 means a 3% chance the result is a fluke." It is the chance of data this extreme if there were no effect, which is not the same thing.
- "p > 0.05 proves no effect." It means the data did not reach the threshold. The test may simply be underpowered.
Practice Questions
Question 1. An A/B test of a new checkout flow shows p = 0.03 against a significance level of 0.05. What can the analyst correctly conclude? A) There is a 3% chance the null hypothesis is true B) The new flow is 97% likely to be better C) Data this extreme would occur about 3% of the time if there were no real difference D) The improvement is big enough to act on
Answer: C. Only C describes the data under the null hypothesis.
Question 2. A test on 4 million sessions finds a 0.01% lift in click-through with p = 0.001. What should the analyst report? A) The lift is highly important because p is very small B) The lift is statistically significant, but its business value must be judged from the effect size and costs C) The result is invalid because the sample is too large D) The null hypothesis has a 0.1% chance of being true
Answer: B. Large samples make small effects significant. Significance and importance are separate questions.
Question 3. A test returns p = 0.12 at alpha 0.05. A stakeholder concludes the change does nothing. Best response? A) Agree; the test proved no effect B) Explain the data did not reach significance; check power and the confidence interval C) Raise alpha until the result is significant D) Rerun the test daily until it is significant
Answer: B. Failing to reject the null is not proof of no effect. C and D are forms of p-hacking.
Related topics
- CBDA statistics questions on hypothesis tests and regression: the wider statistics set
- CBDA analyze data questions
- MVP and experiment design: planning the test before reading it
- CBDA mixed scenario questions
Practice full scenarios on the CBDA practice exam.
FAQ
What does a p-value of 0.03 mean?
If there were no real difference, data at least as extreme as yours would appear about 3% of the time. At a 0.05 significance level, that counts as statistically significant.
Is a p-value the probability that the null hypothesis is true?
No. The p-value is calculated assuming the null is true, so it cannot rate the null's probability. The American Statistical Association states this directly in its 2016 statement.
What is the difference between statistical significance and practical significance?
Statistical significance says the result is unlikely under no effect. Practical significance asks whether the effect is large enough to matter for the business, judged from effect size, confidence intervals and cost.
Sources
- IIBA: Certification in Business Data Analytics (CBDA) — exam format and domain weights, checked October 2026
- IIBA: CBDA Certification Handbook (May 2026)
- IIBA: Introduction to Business Data Analytics, A Practitioner View (2019) — §2.3.4
- American Statistical Association: Statement on p-values (2016)
- NIST/SEMATECH e-Handbook: Critical values and p values
- Spotify Confidence: What is a p-value?
- Statsig: What does statistically significant mean? (2025)
- Evan Miller: How Not To Run an A/B Test (2010)
- Mufaddal Haidermota: Business Data Analytics Certification, Medium (2023)