Inferential statistics: hypothesis testing, Type I and II errors, effect size
Last reviewed September 2026 by Dr. Andrew Wichterman · Aligned to the current NCE blueprint. Valid for exams through June 30, 2027.
AI-narrated overview made with Google NotebookLM. The written lesson below is by Dr. Andrew Wichterman.
Hypothesis testing works exactly like a criminal trial: presume no effect, demand strong evidence, and accept that you can be wrong in two directions. This lesson builds inferential statistics from that one analogy, no math background assumed, and ends with the way the exam phrases every item.
Try to answer each from memory first, even if you have to guess. Guessing before reading makes the answer stick; reveal only after you have tried.
Start here: the courtroom
In a criminal trial the defendant is presumed innocent. The prosecution has to produce enough evidence that a reasonable jury would say "this is very unlikely if he were innocent." If they can, the jury convicts. If not, the verdict is "not guilty," which is not the same as "proven innocent"; it just means there was not enough evidence. And the jury can be wrong in two ways: convict an innocent person, or let a guilty one go.
That is the whole logic of inferential statistics. Replace "innocent" with "the treatment does nothing," "evidence" with "data," and "very unlikely" with "p less than .05," and you have hypothesis testing. Everything in this lesson maps back to the courtroom.
Why we need this at all: samples
You want to know whether a new anxiety program works for anxious adults in general. You cannot test every anxious adult in the world, so you test 40 of them. The 40 are your sample; anxious adults in general are the population. A number that describes the sample (their average GAD-7 score) is a statistic; the same number for the whole population is a parameter. Inferential statistics is the set of rules for deciding what a sample result tells you about the population, given that any sample is a bit off by chance.
The presumption of innocence: the null hypothesis
Every test starts by assuming there is no effect. That assumption is the null hypothesis, written H₀: the treatment group and control group have the same average; the correlation is zero; the two proportions are equal. The researcher's actual prediction, that there is a difference, is the alternative hypothesis, H₁. The whole test is an attempt to gather enough evidence to reject the null, the way a prosecutor tries to overturn the presumption of innocence.
If the evidence is not strong enough, you fail to reject the null. Notice the careful wording. You do not "accept" the null or "prove" there is no effect, any more than a not-guilty verdict proves innocence. You just did not have enough evidence.
How much evidence is enough? p and alpha
Suppose the treatment group averaged 6 points lower than the control group. Could that happen by chance, if the treatment really did nothing? Sure; samples wobble. The question is how often. The p-value is the answer: the probability of getting a difference at least this big by chance alone, if the null were true. A p of .03 means "if the treatment did nothing, you would see a gap this large only 3 percent of the time."
Before running the study, the researcher picks a cutoff for "unlikely enough." That cutoff is alpha, and by convention it is .05 (sometimes .01 for stricter fields). The rule is simple: if p is less than alpha, reject the null and call the result statistically significant. If p is greater than alpha, fail to reject.
The two ways to be wrong
Back to the jury. You can convict the innocent or acquit the guilty. In statistics those two mistakes have numbers for names, and the exam tests them constantly.
A Type I error is rejecting a true null: claiming the treatment works when it does not. Its probability is alpha. If you set alpha at .05, you are accepting a 5 percent chance of a false alarm. A Type II error is failing to reject a false null: missing a treatment that really works. Its probability is called beta. Power is 1 − beta: the probability of detecting a real effect when it is there. Researchers aim for power of .80.
The trade-off
If you make alpha stricter, say .01 instead of .05, you will make fewer false alarms but you will also miss more real effects, because you are demanding stronger evidence. Lower alpha, higher beta, lower power. The exam loves this: any option that says lowering alpha reduces both kinds of error is wrong. To reduce both, you need more participants (a larger n), a bigger real effect, or a more precise measure.
Significant is not the same as important
Here is the trap most students fall into. "Statistically significant" only means the result is unlikely to be chance. It says nothing about how big the effect is. With a huge sample, a difference of half a point on a 63-point scale can be statistically significant and clinically meaningless. With a tiny sample, a large, important difference can fail to reach significance.
Effect size fixes this by measuring how big the difference is regardless of sample size. The most common is Cohen's d: the difference between two group means divided by the standard deviation. Cohen's rough guide: .2 is small, .5 is medium, .8 is large. A d of .8 means the average treated client did better than about 79 percent of the untreated clients. Journals now expect an effect size next to every p-value, and so does the exam.
| Cohen's d | Label | Everyday picture |
|---|---|---|
| 0.2 | Small | Height difference between 15- and 16-year-old girls; you would not notice it in a room |
| 0.5 | Medium | Height difference between 14- and 18-year-old girls; visible if you look |
| 0.8 | Large | Height difference between 13- and 18-year-old girls; obvious at a glance |
One-tailed or two-tailed
If your hypothesis is just "the groups will differ," you have to allow for a difference in either direction, and the .05 is split between the two tails of the distribution: a two-tailed test. If you predict the direction ("treatment will be lower"), all of the .05 sits in one tail: a one-tailed test, which is easier to reach significance with, but only if you were willing to ignore a result in the opposite direction. Two-tailed is the conservative default.
Two families of tests
The specific test you run depends on your data, and lesson 11 walks through choosing one. For now, the two families. Parametric tests assume the data are interval or ratio, roughly normally distributed, and that groups have similar spread. They are more powerful when those assumptions hold: t-tests, ANOVA, Pearson correlation. Nonparametric tests make no such assumptions and work with nominal or ordinal data or badly skewed samples: chi-square, Mann-Whitney U, Wilcoxon, Spearman's rho, Kruskal-Wallis. The rule of thumb: if your data are categories or ranks, or the sample is small and lopsided, go nonparametric.
How the exam asks about this
Define the error. "A researcher concludes a treatment works when it actually has no effect." Type I. "A researcher concludes there is no difference when there really is one." Type II. The false alarm is always Type I; think "I saw something that wasn't there."
Move alpha. "A researcher changes alpha from .05 to .01. What happens?" Fewer Type I errors, more Type II errors, less power. Anything that moves both errors the same way is wrong.
Read a result. "The study reports p = .03." Significant at .05 (reject the null); not significant at .01. "p = .20." Not significant; fail to reject; do not say "proved no effect."
Significance versus size. "A study of 5,000 clients finds a statistically significant but tiny improvement." Ask for the effect size before calling it meaningful. "Which statistic reports the magnitude of a difference independent of sample size?" Effect size, Cohen's d.
Without scrolling up, write everything you remember from this lesson. Then check yourself against the key terms. Recalling the whole lesson once, right after reading it, does more than reading it twice.
Ten more questions from the whole domain, with an explanation after each answer.
{{ doneNote }}