Reliability and validity: every type, in one place, with the distractors.
Test-retest, alternate forms, internal consistency, inter-rater; content, criterion, construct. And the errors the items are built around.
By Dr. Andrew Wichterman · September 15, 2026 · 5 min read · Updated October 4, 2026
Reliability is consistency. Validity is whether the test measures what it claims to. A test can be reliable without being valid, but cannot be valid without being reliable. The exam tests the types, the relationships, and a handful of numbers.
Reliability
- Test-retest: same test, same people, two occasions. Measures stability over time. Sensitive to practice and memory.
- Alternate (parallel) forms: two equivalent versions. Removes memory effects.
- Internal consistency: how items hang together within one administration. Split-half (corrected with Spearman-Brown), Kuder-Richardson (KR-20 for dichotomous items) and Cronbach's alpha (for scaled items).
- Inter-rater (inter-scorer): agreement between observers. Kappa is the statistic.
- The reliability coefficient ranges from 0 to 1; .80 is a common minimum for research, .90 for decisions about individuals. Longer tests are generally more reliable. The standard error of measurement is the spread of an individual's scores around the true score; higher reliability means smaller SEM.
Validity
- Content: the items represent the domain. Judged by experts. Face validity is the weaker cousin: does it look like it measures the thing.
- Criterion-related: the test relates to an outcome. Concurrent if the criterion is measured at the same time; predictive if later. The validity coefficient is the correlation with the criterion.
- Construct: the test measures the theoretical construct. Shown by convergent validity (correlates with measures of the same construct) and discriminant (does not correlate with different constructs). Factor analysis is the usual tool. Modern frameworks treat construct validity as the umbrella.
- Incremental validity: the test adds prediction beyond what is already known.
- Consequential validity: the social consequences of using the test.
The numbers, worked
Spearman-Brown predicts what happens to reliability when a test changes length. Doubling a test with a split-half correlation of .60 gives 2 times .60 over 1 plus .60, which is 1.20 over 1.60, or .75. The formula is the reason the split-half coefficient is corrected: splitting a test in half estimates the reliability of a half-length test, and the correction restores it to full length. The exam tests the direction more than the arithmetic: longer tests are more reliable, other things equal, and adding good items raises the coefficient.
The standard error of measurement is the SD times the square root of one minus the reliability. An SD of 10 and a reliability of .84 give an SEM of 10 times .4, or 4 points, and a 68 percent confidence band of plus or minus 4 around an observed score. A validity coefficient of .50 squared is .25, the coefficient of determination: the test accounts for a quarter of the variance in the criterion. And a ceiling the exam likes: a test's validity cannot exceed the square root of its reliability, so a test with reliability of .64 can have a validity coefficient no higher than .80. Unreliable tests cannot be valid because the ceiling collapses.
Threats to each
Reliability falls when a test is short, when items are ambiguous, when administration varies, when raters drift, and when the sample's range is restricted; correlations shrink when everyone scores similarly, so a test that looks unreliable in a group of graduate students may be fine in the general population. Test-retest coefficients are inflated by memory when the interval is short and deflated by real change when it is long. Internal consistency rises with homogeneous items and falls when a test measures several things at once, which is not always a flaw; a broad test can be useful and have modest alpha.
Validity has its own vocabulary of threats. Construct underrepresentation: the test misses part of the construct, a depression measure with no items on sleep or appetite. Construct-irrelevant variance: the test measures something extra, a math test that requires reading at a level some examinees lack. Criterion contamination: the criterion is influenced by knowledge of the test scores, as when supervisors who saw the aptitude results rate the employees. Restricted range again, since validity coefficients are correlations too. Each of these is a one-line item, and the stem will describe the mechanism without naming it.
Related concepts
- Norms are the reference group. The test is only valid for populations like the norm group.
- Standardization means uniform administration and scoring.
- Bias in a test means it measures differently for different groups; fairness is a broader judgment.
- Ceiling and floor effects limit a test at the extremes.
- Correlation underlies both concepts: reliability coefficients and validity coefficients are correlations.
How items trip people
An item describes two administrations a week apart and asks which reliability: test-retest. It describes two graders: inter-rater. It describes a test predicting first-year grades: predictive validity. It describes experts reviewing items against a syllabus: content validity. It says a test correlates with an established measure given the same day: concurrent. Read for the procedure, then name it.
Three items, worked
A counseling program gives an admissions test and, four years later, correlates the scores with licensure exam results. Predictive validity, because the criterion was measured later; the distractor is concurrent, which would require both to be measured at about the same time. A researcher finds that a new anxiety scale correlates .78 with the Beck Anxiety Inventory and .12 with a measure of extraversion. Convergent evidence from the first correlation and discriminant from the second, together supporting construct validity. The distractor is criterion validity, which would need an outcome rather than another measure of the same construct.
Two clinicians independently score the same twenty Rorschach protocols and their scores agree at kappa .85. Inter-rater reliability, and the distractor is internal consistency, which concerns items rather than scorers. A fourth, for the ceiling: an instrument with a test-retest reliability of .49 is reported to predict job performance at .80. Something is wrong, because the square root of .49 is .70 and the validity coefficient cannot exceed it. The item wants you to notice. Read for the procedure, name the type, and check the number against the ceiling when one is given.
Practice this. The flashcards and practice tests on this site are sorted by content area, so you can drill this one until it holds.
Consistency versus truth. Four ways to check the first, three families to check the second. Read the procedure in the stem and the name follows.