Hypothesis formulation and testing is technique 3.12 of the IIBA Guide to Business Data Analytics: a way to turn a stakeholder's hunch into a statement the data can support or reject [1]. For the CBDA exam, the guide puts the main skill in writing the right hypothesis and reading the result, not in the maths [1]. The guide lists it for Domain 1, Identify the Research Questions, which is 20% of the exam [2].
Hypothesis Formulation and Testing at a glance
Hypothesis Formulation and Testing is a CBDA technique for checking a business claim with a sample of data. The facts:
| Item | Value |
|---|---|
| Guide section | 3.12, chapter 3 (Techniques) [1] |
| Purpose | Turn "intuitive assessments" into "a verifiable and measurable assessment or a research problem" [1] |
| Listed in task | 2.1.7, Select Techniques for Identify the Research Questions (no BABOK reference) (Guide, 2.1.7) |
| Also mentioned in | Exploratory Data Analysis (3.11) and Descriptive and Inferential Statistics (3.9), which refer to hypothesis tests (Guide, 3.11, 3.9) |
| Tests named | t-test, z-test, F-test, chi square test [1] |
| Who runs the test | "A data team conducts these tests", and results are shared with stakeholders [1] |
| Exam | 75 multiple-choice, scenario-based questions in 2 hours; Domain 1 is 20%, Domain 3 is 16% [2] |
Last verified: November 2026 (exam facts on iiba.org; guide text checked section by section).
What the guide says Hypothesis Formulation and Testing is
The guide defines the technique as "a scientific way to verify hunches, intuition, and experience-based decisions" [1]. It is used in business decision-making "to avoid purely empirical decisions", and mainly in problem analysis [1].
Three ideas from 3.12 matter for the exam:
- Samples. The technique is useful when you only have a subset of the population, for example a net promoter score (NPS) survey of some customers [1].
- The analyst's skill is formulation and inference. The guide says the statistical background "can be reviewed through business statistics textbooks", and the analyst's primary skill is to formulate the hypothesis from the business context and read what the test says [1].
- Background topics. Distributions (normal, t, chi square), the central limit theorem, one-tailed and two-tailed tests, and type I and II errors [1].
I like this split. I think this is exactly what a business analyst brings: turning a stakeholder's sentence into a shape that data can answer. The formula is the easier part.
Where the technique is used in the CBDA domains
The guide lists the technique in Domain 1 and connects it to Domain 3 through other techniques.
| Domain or task | What the guide says |
|---|---|
| 2.1.7, Identify the Research Questions | Used "to develop a premise" from a stakeholder's or SME's opinion, "statistically justified through the data to formulate research questions accurately" (Guide, 2.1.7) |
| 2.1.3, Assess Current State | A leader's claim that internet sales fell because of low engagement is a hypothesis; analysts look for "concrete evidence from data and analysis" (Guide, 2.1.3) |
| 3.11 EDA, Analyze Data | EDA comes "before applying more formal approaches such as hypothesis testing"; one EDA step is "Build and test a basic hypothesis to test insights" (Guide, 3.11) |
| 3.9 Statistics | "A z-test deduces what range an observed sample mean can take"; a t-test is used instead when there are fewer observations (Guide, 3.9) |
| 3.17 Surveys | Good question design starts with "understanding the key hypothesis to be tested" (Guide, 3.17) |
If you want the Domain 1 picture first, see Identify the Research Questions. For Domain 3, see Analyze Data.
The seven-step hypothesis testing process
The guide simplifies the process into seven steps (3.12.3) [1]:
- State the hypothesis.
- Select appropriate test statistics.
- Specify the level of significance.
- State the decision rule regarding the hypothesis.
- Collect samples and calculate the sample statistics.
- Make a decision regarding the hypothesis.
- Provide insights based on the results of the test.
Two details from the guide's terms list are worth knowing. The null hypothesis is "the default position of a claim that results in no change of the current state" [1]. A one-tailed test looks at only one side of the curve, for example when a baseball bat longer than 42 inches is not allowed [1]. Statistics courses add one convention: "the null hypothesis always includes the equality" [3].
Step 1 is where most business cases go wrong, so here is how stakeholder sentences become hypotheses. The sentences are my examples, not the guide's:
| Stakeholder says | H0 (no change) | H1 (the claim) | Tail |
|---|---|---|---|
| "Customers wait less than 3 days now" | Mean delivery time ≥ 3 days | Mean delivery time < 3 days | Lower |
| "The new checkout changed conversion" | Conversion is the same | Conversion is different | Two |
| "App users buy more often" | App users buy as often as others | App users buy more often | Upper |
Worked example: the guide's bank case, step by step
The guide's own example is a lower-tailed z-test [1]. A bank surveys 100 customers: average weekly visits are 4.5 with a standard deviation of 2. The banker believes the true average is above 5. A separate study recommends closing 50% of teller windows if the average is below 5 [1].
| Step | Bank case |
|---|---|
| 1. Hypothesis | H0: "The average weekly visit is greater than 5", the banker's belief [1]; in textbook notation H0: mean ≥ 5, because the null includes the equality [3]. H1: below 5 |
| 2. Test statistic | z-test, 100 observations [1] |
| 3. Significance | 5%, so 95% confidence [1] |
| 4. Decision rule | Reject H0 if z is below the critical value of about -1.65 [1] (statistics courses use -1.645 [4]) |
| 5. Calculate | Standard error = 2 / √100 = 0.2; z = (4.5 - 5) / 0.2 = -2.5 [1] |
| 6. Decide | -2.5 is below -1.65, so H0 is rejected [1] |
| 7. Insight | The banker's assumption of more visits "is statistically not sound" [1] |
Two numbers the guide does not give, which I calculated from the same data. The one-tailed p-value for z = -2.5 is about 0.006. A 95% confidence interval for the mean is 4.5 ± 1.96 × 0.2, so about 4.11 to 4.89 visits a week. Both point the same way as the z-test: the average is below 5. Strictly, a one-tailed test at 5% pairs with a 90% interval, about 4.17 to 4.83, which also stays below 5.
The guide's table then ends with two lines that are easy to misread. One says "The recommendation of closing 50% teller window should not be considered" [1]. The other says analysts could use the result "to look for more information", for example pay days, when visits are higher [1]. My reading for the exam: the test rejects the banker's hunch, but it does not make the decision. The next step is more information, and that matches the first limitation in 3.12: a test "is not a replacement for any analysis led conclusions or decisions" [1].
Strengths and limitations of hypothesis testing
The guide gives five strengths and four limitations (3.12.4) [1].
| Strengths | Limitations |
|---|---|
| Focuses analysis on one business problem without drifting into secondary details | Shows whether a claim is statistically sound; does not replace analysis-led conclusions |
| Limits the data needed to verify a claim | May ignore signals that exploratory analysis would find |
| With correct formulation, needs fewer assumptions about the distribution | The process and results may be hard to communicate; use simple examples |
| Advanced measures (F score, p values) show which factors influence an outcome | Results are probabilistic; state confidence intervals and significance levels |
| Results feed further elicitation for decision-making |
I think the communication limitation matters most in practice. A p-value does not measure the size or importance of an effect [5], yet "significant" easily sounds like "important". The guide's advice is to explain with simple examples [1], and two authors from a statistics consultancy suggest saying "in our sample, the lift was 10%" instead of "this feature will increase revenue by 10%" [6].
Where the guide simplifies the statistics
The guide warns that its explanations "are neither comprehensive nor exhaustive" [1]. For the exam, answer in the guide's terms. For work, know where the textbook wording is more precise:
| Topic | Guide wording | Statistics references |
|---|---|---|
| Confidence interval | A level such as 95% through which "a likelihood of the result being correct can be established" [1] | 95% describes the method: intervals built this way bracket the true parameter in about 95% of cases [7]; the population mean "is a constant, not a random variable" [8] |
| Standard error | Approximates the true standard deviation by dividing by the square root of the number of observations [1] | "The standard error is the standard deviation of a sampling distribution" [9]; for a mean, SE = s / √n [10] |
| p-value | The probability of observing a result "purely by chance when the null hypothesis is correct" (Guide, 3.18 sidebar) | p-values do not measure the probability that the hypothesis is true, or that data came from chance alone [5] |
| Errors | One-tailed and two-tailed tests with type I and II errors [1] | Type I: rejecting a true H0, with probability alpha, commonly .05 [11]; type II: failing to reject a false H0 [11] |
| Critical value | -1.65 at 5%, one tail [1] | -1.645 [4] |
None of this changes the bank case result. It changes how you explain it. Our separate article on what a p-value really means goes deeper into the p-value row.
What happens when companies test their hunches
Real experiments show that most business hunches are wrong. That is the strongest argument for the technique.
- At Google and Bing, "only about 10% to 20% of experiments generate positive results"; at Microsoft about a third of tested ideas worked [12].
- Booking.com runs a little over 1,000 concurrent experiments, and a business school professor who studied it describes a company where about "9 out of 10 things fail" [13]. The same professor estimates that companies may find only about 10% of experiments improve the bottom line [14].
- A 2023 review by industry and academic statisticians says "more than 50% of ideas fail to generate meaningful improvements" [15].
Practitioners add the human side:
- A senior manager's hunch lost. An early Amazon engineer built shopping cart recommendations. A marketing senior vice-president feared they would distract buyers, and he was "forbidden to work on this any further". He ran a test anyway, and the feature won by a wide margin [16].
- The highest-paid opinion. An experimentation leader describes a tax season at Intuit TurboTax: of 40 experiments, 38 did not win [17].
- Peeking. An Airbnb data scientist describes a price filter test where the p-value crossed 0.05 after 7 days with an apparent 4% effect. Run longer, the effect was neutral [18]. A writer on A/B testing statistics shows that in a worst case, where significance is checked after every observation and the test stops as soon as it looks significant, the false positive rate can reach 26.1% instead of 5% [19].
- Sample size. Two UX research methodologists calculate that about 75 respondents (90% confidence, 50% power) can detect a 30-point NPS difference, while a 3-point difference needs more than 23,000 (95% confidence, 80% power) [20]. That is the guide's NPS example in numbers.
- Intuition first, data second. A former BA turned data scientist describes a bank BA whose idea was to combine ATM, online query and complaint data to find duplicate customer contacts and cut costs. Intuition set the hypothesis, data had to check it [21].
- Consulting. An ex-Bain consultant names hypothesis-driven thinking as the skill that changed his work most: "I think the problem is X, here's how we test it." [22]
From the exam side, candidates describe CBDA questions as "scenario-based" [23]. I found no candidate report that names hypothesis testing questions specifically, so I would not expect heavy calculation. Expect scenarios where a stakeholder's belief needs to be checked.
Experts disagree on significance thresholds
Statisticians agree that a p-value alone is a weak basis for a decision, but not on what to do about it.
1. Retire "statistically significant". More than 800 scientists signed a call to stop sorting results into significant and not significant, while keeping p-values [24].
2. Keep a threshold, make it stricter. A group of 72 researchers proposed moving the default threshold for new discoveries from 0.05 to 0.005 [25].
3. Justify your own alpha. A reply signed by 88 researchers argues there should be no universal threshold: choose alpha for each study, given the costs of errors [26].
4. Significance tests remain useful. A 2021 American Statistical Association task force wrote that p-values and significance tests, properly applied, "are important tools" [27].
5. Exploration matters as much as confirmation. A classic paper argues that "finding the question is often more important than finding the answer" [28].
For the CBDA, the guide sits closest to positions 4 and 5. It uses fixed levels such as 95% or 99%, and it says hypothesis tests may miss what exploratory analysis would find [1]. Position 3 is the one I find most practical in business: a 5% error rate on a cheap A/B test is not the same risk as on closing half the teller windows.
Hypothesis testing vs similar CBDA techniques
Several chapter 3 techniques sound alike. The difference is in what each starts from.
| Technique | Starts from | Output |
|---|---|---|
| Hypothesis Formulation and Testing (3.12) | A stated claim or hunch | Reject or do not reject H0, with a confidence level [1] |
| Exploratory Data Analysis (3.11) | Data, "without any prior assumption or bias" | Patterns, preliminary insights, refined research problems (Guide, 3.11) |
| Descriptive and Inferential Statistics (3.9) | A population or a sample | Summaries and "informed generalizations" (Guide, 3.9) |
| Problem Shaping and Reframing (3.15) | A problem statement | An "alternative formulation of the problem" (Guide, 3.15) |
For a related exam topic, see correlation vs causation: a significant test does not prove a cause.
Exam traps for Hypothesis Formulation and Testing
| Scenario | Tempting answer | Guide reading |
|---|---|---|
| A manager is sure of a cause, based on experience | Accept it as the research question | Turn the opinion into a premise and justify it statistically (Guide, 2.1.7) |
| The analyst is asked to run the test | The analyst must do the maths | Formulate and interpret; the data team tests (3.12.2) |
| The test rejects a stakeholder's claim | Act on the result immediately | Tests do not replace analysis-led decisions; look for more information [1] |
| A result is "significant" | Report it as certain | Results are probabilistic; state confidence and significance levels [1] |
| Little is known about the data yet | Start with a formal test | EDA first (3.11.1) |
| Stakeholders do not follow the test | Show the formulas | Use simple examples to build confidence [1] |
Check yourself: CBDA practice questions
1. A sales director insists that lower web sales come from weak customer engagement. What should the analyst do first? A) Build a campaign to raise engagement. B) Formulate the claim as a hypothesis and test it with data. C) Reject the claim as opinion. D) Ask the data team for a churn model.
Answer: B. The guide uses the technique to turn a stakeholder's opinion into a premise justified through data (Guide, 2.1.7). C ignores useful domain knowledge.
2. In the guide's bank case, z = -2.5 and the critical value is -1.65. What follows? A) H0 is not rejected. B) H0 is rejected, so the banker's belief is not supported. C) The teller windows must close. D) The sample is too small.
Answer: B. -2.5 is below -1.65 [1]. C is wrong because the test alone does not make the decision.
3. Which statement about the analyst's role is in line with the guide? A) The analyst must calculate all test statistics. B) The analyst's main skill is formulating the hypothesis and interpreting the result. C) The analyst picks the significance level alone. D) Hypothesis tests replace exploratory analysis.
Answer: B. Per 3.12.2 [1]; D contradicts 3.12.4.
Conclusion: the gap between the guide and real experiments
The guide teaches the right habit, test the hunch before acting, with simplified statistics. Practice adds sample size, peeking and a lot of failed ideas.
| Area | Guide | Reality | Delta |
|---|---|---|---|
| Who runs the test | Data team; analyst formulates and interprets [1] | Experimentation teams run online tests at scale, thousands a year [12, 13] | No gap |
| How often hunches hold | Not covered | Most ideas fail; 10-20% positive at Google and Bing [12] | Added by employers |
| Confidence wording | "Likelihood of the result being correct" [1] | 95% describes the method, not one interval [7] | Reduced to a simpler wording |
| Stopping rules | Not covered | Peeking inflates false positives [19] | Added by employers |
| Sample size | Samples may not be accurate for the whole base [1] | Small differences need very large samples [20] | Under-used |
For the exam, learn the guide's seven steps, terms and limitations. For work, add sample size and a fixed stopping rule. Practice both with the CBDA statistics questions, the weekly question breakdown on a VP's sales hunch and full mocks in the CBDA exam simulator.
FAQ
What is hypothesis formulation and testing in the CBDA guide?
It is technique 3.12 of the IIBA Guide to Business Data Analytics. It turns stakeholder hunches into hypotheses that a statistical test on sample data can reject or not reject. The guide lists it for the Identify the Research Questions domain.
Do I need to calculate z-tests on the CBDA exam?
The guide says a data team conducts the tests and the analyst's primary skill is formulation and interpretation. Knowing how a z-score, a critical value and a standard error work helps you read scenario questions, such as the guide's bank example.
What is the null hypothesis in business terms?
The guide defines it as the default position of a claim that results in no change of the current state. In the bank case, the null is the banker's belief that average weekly visits are greater than 5.
What is the difference between hypothesis testing and exploratory data analysis?
Exploratory data analysis looks at data without prior assumptions to find patterns. Hypothesis testing starts from a stated claim and checks it. The guide places EDA before formal approaches such as hypothesis testing.
Does a rejected null hypothesis tell the business what to do?
No. The guide says hypothesis tests are not a replacement for analysis-led conclusions or decisions. In the bank case, the result points to looking for more information before acting.
Sources
- IIBA: Guide to Business Data Analytics, 3.12 Hypothesis Formulation and Testing
- IIBA: Certification in Business Data Analytics (CBDA)
- Penn State STAT 200: Writing hypotheses
- Penn State STAT 415: Tests about one mean
- American Statistical Association: Statement on p-values (2016)
- Statsig: How to report test results (2025)
- NIST/SEMATECH e-Handbook: What are confidence intervals?
- Penn State STAT 415: Interpretation of a confidence interval
- Penn State STAT 200: Sampling distributions
- Penn State STAT 200: One sample mean
- Penn State STAT 200: Type I and type II errors
- Kohavi and Thomke: The surprising power of online experiments (HBR, 2017)
- HBS Working Knowledge: At Booking.com, innovation means constant failure
- HBS Working Knowledge: Creating the experimentation organization (2019)
- Larsen et al.: Statistical challenges in online controlled experiments (2023)
- Geeking with Greg: Early Amazon, shopping cart recommendations (2006)
- Statsig: How culture drives successful experimentation (2024)
- Airbnb Engineering: Experiments at Airbnb (2014)
- Evan Miller: How not to run an A/B test (2010)
- MeasuringU: Sample sizes for comparing Net Promoter Scores (2021)
- Analyst's Corner: The imperative of intuition + data for better business analysis (2020)
- PrepLounge consulting forum: Which consulting skill improved your effectiveness (2026)
- Techcanvass: CBDA certification journey of Aniket (2024)
- Amrhein, Greenland, McShane: Retire statistical significance (Nature, 2019)
- Benjamin et al.: Redefine statistical significance (2018)
- Lakens et al.: Justify your alpha (2018)
- ASA President's Task Force statement on statistical significance and replicability (2021)
- Tukey: We need both exploratory and confirmatory (1980)