Survivorship bias is the error of drawing conclusions only from the cases that made it through a filter, while the cases that failed are missing from the data. On the IIBA Certification in Business Data Analytics (CBDA) exam it usually appears as a sampling question: a churn study that surveys only active customers. The fix is to ask who is missing from the data before judging anything else.
Why survivorship bias matters on the CBDA exam
Survivorship bias is a data bias problem, and data bias is one of the quality checks the IIBA expects analysts to run. It touches Source Data (15% of questions) and Analyze Data (16%), and it decides whether a report in Interpret and Report Results (20%) can be trusted.
| CBDA fact | Value |
|---|---|
| Questions | 75, multiple-choice, scenario-based |
| Time | 120 minutes |
| Pass mark | Not published; pass or fail shown on screen |
| Source Data | 15% of questions |
| Analyze Data | 16% of questions |
Last verified: October 2026 on the IIBA CBDA page and the CBDA Certification Handbook (May 2026).
How the IIBA frames it
IIBA's public Introduction to Business Data Analytics: A Practitioner View (2019, §2.3.3) lists four checkpoints for exploring data: integrity, validity, reliability and bias. The bias checkpoint asks: "Does the data portray an accurate picture of a given situation?" The document's example is a survey whose participants are not a representative sample of the population.
The same document defines sampling as working with a representative subset of a larger population (§2.2.2) and describes weighting as a way to correct uneven representation of subgroups (§2.3.2). Weighting helps when a group is under-represented. It cannot help when the group is absent. That is what survivorship bias does: the sampling frame excludes the cases that hold the answer.
The textbook case is Abraham Wald's work for Columbia's Statistical Research Group in World War II. Officers wanted to armor the parts of returning bombers that showed bullet holes. Wald argued for armor where returning planes had no holes, because planes hit there did not come back. As a 2026 Vintage Aviation News article puts it, "The only aircraft that could be inspected were those that survived and returned home."
Worked example: the churn survey
A subscription business wants to know why customers leave. The analyst surveys all current active customers about satisfaction drivers and gets 4,000 responses.
The sampling frame is "customers who stayed." Every respondent is, by definition, someone the product kept. Their answers describe what retains loyal customers, not what drives others away. Stronger statistics on this sample do not fix it.
A sound design starts from the population of interest:
- Define the population. Customers who cancelled in the last two quarters, plus a comparison group who stayed.
- Collect from churned customers. Exit surveys at cancellation, win-back interviews, support tickets and usage logs before cancellation.
- Use passive data. Behavioral data such as login frequency and feature use exists for churned customers even when they will not answer a survey.
- Compare cohorts. Report churned and retained cohorts side by side instead of blending them into one average.
Why each wrong option fails
The original question asks for the biggest concern with surveying active customers to understand churn.
| Option | Why it is tempting | Why it fails |
|---|---|---|
| A. Sample size is too small | Size is a common concern | The stem gives no size; and size does not fix a broken frame |
| B. Data bias: churned customers are excluded | Less obvious | Correct: the population that holds the answer is missing |
| C. Satisfaction cannot be quantified | Sounds rigorous | False; satisfaction is measured routinely with scales |
| D. Surveys are self-reported | A real limitation | Secondary; it applies to any survey, while this frame fails by design |
The trap is D. It is true, so it feels safe. But the question asks for the biggest concern, and a missing population outranks noisy answers.
What practitioners say
SaaS leaders hit the same bias in metrics, not only surveys. In a 2015 Kellblog post, Dave Kellogg showed how computing expansion only from customers still on the books inflates the rate. In his example the biased method gave 111% and the cohort-based method 71%. His verdict: "Calculating it the first way can be misleading."
Customer research teams describe the survey version. Koji's guide to survivorship bias in customer research recommends interviewing churned customers as a standing practice, segmenting results by lifecycle status, and reporting cohorts separately. It also notes that non-response bias compounds the problem, because unhappy customers are less likely to reply.
LN Mishra of Adaptive US, a CBDA holder, ranks data interpretation as a high-priority study topic in his 2024 exam guide. Spotting who is missing from a dataset is part of that skill.
Common misconceptions
- "A bigger sample fixes bias." A larger biased sample gives a more precise wrong answer.
- "Weighting can always correct it." Weighting rebalances groups that are present. It cannot invent a group that was never sampled.
- "Survivorship bias is only about success stories." It applies to any filter: active users, completed orders, closed tickets, funded startups.
Practice Questions
Question 1. An analyst wants to understand why customers churn. They survey all current active customers about satisfaction drivers. What is the biggest concern? A) The sample size is too small B) Data bias: the sample is not representative because churned customers are excluded C) Satisfaction cannot be quantified D) Surveys are self-reported data
Answer: B. The people who hold the answer are outside the sampling frame.
Question 2. A bank studies loan defaults using only loans that are still open today. Which problem is most likely? A) Defaulted loans that were written off are missing, so default risk is understated B) Open loans have too many variables C) The data is too recent to analyze D) Loan data cannot be sampled
Answer: A. Written-off loans left the dataset, so the remaining loans look safer than the full portfolio.
Question 3. Which action best reduces survivorship bias in a churn analysis? A) Increase the number of active customers surveyed B) Add usage and support data for customers who cancelled, and compare them with retained customers C) Switch from a survey to a focus group of loyal customers D) Apply survey weights to the active-customer responses
Answer: B. It brings the missing population back into the data. A, C and D still sample only survivors.
Related topics
- Sourcing and validating data for CBDA
- CBDA source data questions on collection plans and sampling
- CBDA data quality questions
- CBDA analyze data questions on preparation and exploration
- CBDA study plan: where Source Data fits in six weeks
For more scenario drills, try the CBDA practice exam.
FAQ
What is survivorship bias in simple terms?
It is judging a group by the members who passed a filter and ignoring those who did not. Studying only returning planes, active customers or surviving companies leads to conclusions that do not hold for the whole population.
Is survivorship bias on the CBDA exam?
The term may not appear, but the concept does. IIBA lists data bias as a checkpoint when exploring data, and scenario questions test whether a sample represents the population the research question is about.
How is survivorship bias different from non-response bias?
Survivorship bias excludes a group from the sampling frame entirely. Non-response bias happens when a group is in the frame but answers less often. Both distort results, and they often occur together.
Sources
- IIBA: Certification in Business Data Analytics (CBDA) — exam format and domain weights, checked October 2026
- IIBA: CBDA Certification Handbook (May 2026)
- IIBA: Introduction to Business Data Analytics, A Practitioner View (2019) — §2.2.2, §2.3.2, §2.3.3
- Vintage Aviation News: Abraham Wald's counterintuitive armor theory (2026)
- Kellblog: Survivor bias in churn calculations, Dave Kellogg (2015)
- Koji: Survivorship bias in customer research
- Adaptive US: Mastering the CBDA exam, LN Mishra (2024)