Reliability and validity: every type the exam names
Last reviewed September 2026 by Dr. Andrew Wichterman · Aligned to the current NCE blueprint. Valid for exams through June 30, 2027.
AI-narrated overview made with Google NotebookLM. The written lesson below is by Dr. Andrew Wichterman.
If you have never taken a statistics or testing course, or took one years ago and remember none of it, start here. This lesson explains reliability and validity from scratch, with everyday examples and pictures, and then shows you exactly how the exam asks about them.
Try to answer each from memory first, even if you have to guess. Guessing before reading makes the answer stick; reveal only after you have tried.
Start here: two questions about any test
Suppose you weigh yourself every morning. Two different things could go wrong with your scale. It could give you a different number every day even though your weight hasn't changed. Or it could give you the same number every day, but that number could be wrong. Those two problems have names, and the whole lesson is about them.
Reliability means the test gives the same answer each time. Same person, same test, same result. Think "re-liable," you can rely on it to repeat itself.
Validity means the test gives the right answer. It actually measures what it says it measures. A depression scale that really picks up depression, not just tiredness.
The dartboard picture
The easiest way to keep the two ideas apart is to picture someone throwing darts. Where the darts land tells you about the test.
Where the "wrong" part of a score comes from
No test score is perfect. Part of a client's score on a depression questionnaire is their actual level of depression. Part of it is things that have nothing to do with depression: they were tired, they misread a question, the room was noisy, they guessed on one. Testing people call this extra part "error." It doesn't mean anyone made a mistake. It just means noise got mixed in.
Reliability is reported as a number between 0 and 1. A reliability of .90 means about 90 percent of the differences between people's scores are real differences, and about 10 percent is noise. For tests used to make decisions about individual people, you want .90 or higher. For research on groups, .70 is often accepted.
The four kinds of reliability
"Consistent" can mean consistent across different things. Consistent over time? Consistent between two versions of a test? Consistent between the questions inside one test? Consistent between two people scoring it? Each one is a separate type, and the exam asks you to name the type from a short description. The table gives you the everyday version of each.
| Type | Consistent across… | Everyday version | How you spot it on the exam |
|---|---|---|---|
| Test-retest | Time | Take the test today and again in two weeks. Do you get about the same score? | Words like "stability," "over time," "two weeks later" |
| Alternate forms (also "parallel forms") | Two versions of the test | Form A and Form B are supposed to be interchangeable. Do people score about the same on both? | "Two versions," "Form A and Form B," "equivalent" |
| Internal consistency | The questions inside one test | If a test has 20 questions about anxiety, do people who say yes to one tend to say yes to the others? | "Items hang together," "one sitting," "Cronbach's alpha," "split-half" |
| Inter-rater (also "inter-scorer") | Different people doing the scoring | Two counselors watch the same session and rate the client's anxiety. Do they give similar ratings? | "Two observers," "judges agree," "kappa" |
The statistics you only need to recognize
You will not have to compute any of these. You only need to match the name to the job.
| Name | What it is for |
|---|---|
| Cronbach's alpha | Internal consistency when the questions have a scale of answers (strongly agree to strongly disagree) |
| KR-20 (Kuder-Richardson) | Internal consistency when each question is simply right or wrong |
| Split-half | Internal consistency by splitting one test into two halves (odd questions vs. even questions) and comparing them. It underestimates the real reliability because each half is only half as long, so it gets corrected with the Spearman-Brown formula |
| Cohen's kappa | Agreement between two raters, adjusted for the agreement they would get by chance |
Things that make a test more or less reliable
Longer tests are more reliable, because with more questions the lucky guesses and careless slips average out. That is the same reason a 100-question exam tells you more than a 10-question quiz. A test given to a wide mix of people looks more reliable than the same test given to a very similar group, because there are bigger real differences to detect. And speeded tests (the kind you cannot finish) should not be judged by internal consistency, because the blank answers at the end make the questions look inconsistent when they are not.
One person's score: the standard error of measurement
Reliability describes a test. But a counselor usually cares about one client. If a client scores 100 on an IQ test, what is their real IQ? Not exactly 100; that score has noise in it. The standard error of measurement, usually written SEM, tells you how far off a single person's score is likely to be. Think of it as a margin of error, like the "plus or minus 3 points" in a poll.
The connection to reliability is simple: the more reliable the test, the smaller the SEM. A perfectly reliable test would have an SEM of zero. You may see the formula SEM = SD × √(1 − reliability). You do not need to calculate it; you need to know that higher reliability means a narrower range around the score.
The kinds of validity
Validity is the harder idea, because "does the test measure the right thing?" can be checked in several different ways. Each way of checking has a name. The good news is that they sort into three families.
Content validity: does it cover the right material?
Imagine a final exam for a statistics course that only asks about the first chapter. It might be a fine test of chapter one, but it is not a valid test of the course. Content validity asks whether the questions cover everything the test is supposed to cover, in the right proportions. It is judged by experts reading the questions, not by a statistic. The NCE itself is checked this way: the number of questions in each domain is matched to the blueprint percentages.
Criterion validity: does it match something real?
Here you compare the test to some outside standard, called the criterion. There are two versions, and the only difference is when the criterion is measured.
Concurrent validity: the criterion is measured at about the same time. A new, short depression questionnaire is given the same week a psychologist interviews the same clients. If the questionnaire agrees with the interviews, it has concurrent validity. The word to look for is "now."
Predictive validity: the criterion is measured later. GRE scores are compared to how students actually do in graduate school a year later. If high scorers do better, the GRE has predictive validity. The words to look for are "later," "future," and "predict."
Construct validity: does it behave like the idea it measures?
A "construct" is just an idea we cannot see directly, like anxiety, intelligence or self-esteem. You cannot hold anxiety in your hand, so how do you know a test measures it? You check whether the test behaves the way anxiety should behave. Scores should go up when people are anxious, come down after treatment, and line up with other anxiety measures. This is the broadest kind of validity, and the other kinds are really evidence for it.
Two pieces of construct validity get their own names. Convergent validity: the test agrees with other tests of the same thing (two anxiety scales correlate strongly). Discriminant validity, sometimes called divergent: the test does not agree with tests of unrelated things (the anxiety scale barely correlates with a reading test). You want both: it should converge with its cousins and diverge from strangers.
Face validity: the one that doesn't count
Face validity means the test looks like it measures what it claims, to the person taking it. A depression scale that asks about sadness and sleep has face validity. This is not real evidence of validity; a test can look right and be wrong, or look strange and work well. Face validity only matters because people cooperate more with a test that seems sensible. If an exam option offers face validity as proof that a test works, it is a wrong answer.
How the two connect
Go back to the dartboard. You can throw a tight cluster in the wrong place: reliable, not valid. But you cannot hit the bullseye consistently with a scattered throw. In testing terms: a test can be reliable without being valid, but it cannot be valid without being reliable. Reliability is the floor that validity stands on.
There is even a limit you can put a number on. The validity of a test can never be higher than the square root of its reliability. A test with reliability .64 can have validity of at most .80 (because the square root of .64 is .80). You will not be asked to compute this, but you may be asked which one is "necessary" for the other. The answer is that reliability is necessary for validity.
Things that make the numbers lie
The exam names a handful of situations that distort reliability or validity results. Each appears as a one-line item, so the everyday version is enough.
| Name | What is going on | Everyday example |
|---|---|---|
| Restriction of range | You only studied people who are all similar, so the test can't show how well it separates people | Checking whether a hiring test predicts job performance, but only among the people who were hired (the low scorers were never hired, so they aren't in the data) |
| Criterion contamination | The person judging the outcome already knows the test scores, so the two agree more than they should | A supervisor rates an employee's performance after seeing the employee scored high on the aptitude test |
| Regression toward the mean | People with very extreme scores tend to score closer to average the next time, whether or not anything changed | The students with the lowest pretest scores show the biggest "improvement," even with no tutoring |
| Attenuation | Noise in either measure weakens the correlation between them, making a valid test look weaker than it is | A test with modest reliability shows a validity of .40; corrected for the noise, the real relationship is stronger |
| Cross-validation | Checking whether a result found in one group holds up in a second group | A formula that predicts treatment dropout at one clinic is tested at a different clinic |
How the exam asks about this
Almost every reliability and validity item is one of three kinds.
1. Definition to name. "The degree to which a test gives consistent results when given twice to the same people" is test-retest reliability. Learn the everyday version of each term and these are free points.
2. Story to type. "A school counselor compares students' SAT scores to their college GPA two years later." The word "later" makes this predictive validity. "Two supervisors independently rate the same trainee's session" is inter-rater reliability.
3. The relationship. "Which is necessary for the other?" Reliability is necessary for validity. "Can a test be reliable but not valid?" Yes. "Valid but not reliable?" No.
When two options both seem right, find the time word. Now, same day, same week: concurrent. Later, future, predict: predictive. Over time, stability, two administrations: test-retest. And if an option says a test is valid because it "looks like" it measures the trait, that is face validity and it is the wrong answer.
Report a problem with this lessonWithout scrolling up, write everything you remember from this lesson. Then check yourself against the key terms. Recalling the whole lesson once, right after reading it, does more than reading it twice.
Ten more questions from the whole domain, with an explanation after each answer.
{{ doneNote }}