Research designs and threats to internal and external validity
Last reviewed September 2026 by Dr. Andrew Wichterman · Aligned to the current NCE blueprint. Valid for exams through June 30, 2027.
AI-narrated overview made with Google NotebookLM. The written lesson below is by Dr. Andrew Wichterman.
A counselor's new group seems to help. Did it? Research design is the set of tricks for ruling out everything else that could explain the change. This lesson explains the vocabulary, the three kinds of study, the threats that fool researchers, and how the exam tests each, assuming no prior research course.
Try to answer each from memory first, even if you have to guess. Guessing before reading makes the answer stick; reveal only after you have tried.
Start here: does the treatment work, or does it just look like it works?
A counselor runs a new anxiety group. Eight weeks later the members report less anxiety. Did the group help? Maybe. Or maybe the members would have improved anyway with time; maybe the most motivated clients were the ones who signed up; maybe they just learned to answer the questionnaire differently the second time. Research design is the set of tricks for ruling out the "maybes" so that you can say the treatment, and not something else, produced the change. The exam does not ask you to run studies. It asks you to name the parts, spot the design, and recognize what could be fooling the researcher.
The words you need first
| Term | Plain meaning | In the anxiety-group example |
|---|---|---|
| Independent variable (IV) | The thing the researcher changes or applies; the suspected cause | The anxiety group (getting it or not) |
| Dependent variable (DV) | The thing the researcher measures; the outcome that "depends" on the IV | Anxiety score after eight weeks |
| Treatment (experimental) group | The people who get the IV | Clients in the group |
| Control group | The people who do not get it, for comparison | Clients on the waiting list |
| Hypothesis | The specific prediction being tested | "Group members will report lower anxiety than the wait list" |
| Operational definition | Exactly how a fuzzy idea will be measured | "Anxiety" = score on the GAD-7 |
| Confounding (extraneous) variable | Something else that differs between the groups and could explain the result | Group members also happened to start medication |
| Placebo | A fake treatment that controls for the effect of expecting to improve | An "attention only" group that meets but does no anxiety work |
| Double-blind | Neither participants nor the people measuring them know who got the real treatment | Rare in counseling, common in drug trials |
The single most important idea: random assignment
The worry in every study is that the two groups were different before the treatment started. If the most motivated clients joined the group and the least motivated stayed on the wait list, the group will look effective even if it did nothing. Random assignment fixes this. You take everyone who agreed to be in the study and use a coin flip (or a random number list) to decide who gets the treatment. Now the only systematic difference between the groups is the treatment itself, and any pre-existing differences are spread evenly by chance.
Do not confuse this with random selection, which is about how you got your participants in the first place. Random selection from a population lets you generalize your result to that population. Random assignment to groups lets you claim the treatment caused the result. A study can have one without the other. Most counseling studies have neither, which is why so few can say "caused."
Three kinds of study
A true experiment has random assignment and a manipulated IV. It is the only design that supports a causal claim. A quasi-experiment manipulates the IV but uses existing groups without random assignment: a new curriculum in one school compared to the old one in another. It is often the best a counselor can do in the real world, and it can be strengthened with a pretest to show the groups started out similar. A correlational study manipulates nothing; it measures two variables in the same people and computes the relationship. It can show that anxiety and sleep problems go together but cannot say which causes which, or whether a third thing causes both.
Other designs the exam names
| Design | What it is | Example |
|---|---|---|
| Pretest-posttest control group | Measure both groups before and after; the classic experimental design | GAD-7 before and after, group vs. wait list |
| Posttest-only control group | Measure only after; relies on random assignment to make the groups equal at the start | Same, without the pretest |
| Solomon four-group | Combines the two above to detect whether taking the pretest itself changed people | Four groups: with/without pretest × with/without treatment |
| Single-subject (N = 1) designs: ABAB, multiple baseline | One person measured repeatedly; treatment is applied, withdrawn, reapplied | Track a child's outbursts for two weeks (A), add a reward system (B), remove it (A), add it back (B) |
| Longitudinal | Same people followed over a long time | Track a cohort from high school to age 30 |
| Cross-sectional | Different age groups measured at one time | Compare 20-, 40- and 60-year-olds today |
| Ex post facto (causal-comparative) | Groups formed by something that already happened; looks backward | Compare adults who were and were not bullied as children |
| Survey / descriptive | Describe what is, without comparison groups | What percentage of counselors use telehealth? |
Two kinds of "is this study any good?"
Here "validity" means something different from the test validity in lesson 4, though the spirit is the same: is the conclusion trustworthy?
Threats to internal validity: the things that fool you
Campbell and Stanley's list of threats is heavily tested. Each is a reason a change might happen that has nothing to do with the treatment.
| Threat | Plain meaning | Example |
|---|---|---|
| History | An outside event happened during the study | A community tragedy raises everyone's anxiety mid-study |
| Maturation | People change on their own with time | Children's attention naturally improves as they age |
| Testing | Taking the pretest changes how people do on the posttest | Clients remember the questions and answer "better" |
| Instrumentation | The measuring tool or the rater changed | A new version of the questionnaire is used at posttest; the observer gets stricter |
| Statistical regression | People picked for extreme scores drift back toward average on their own | The most anxious clients improve most, treatment or not |
| Selection | The groups were different to begin with | Volunteers vs. non-volunteers |
| Mortality (attrition) | People drop out, and the ones who stay are different | The clients who were not improving quit the group |
| Diffusion of treatment | The control group picks up the treatment informally | Wait-list clients hear about the techniques from friends in the group |
| Experimenter (Rosenthal) effect | The researcher's expectations leak to participants | Warmer with the treatment group |
| Hawthorne effect | People change simply because they know they are being studied | Attendance improves because someone is counting |
| Demand characteristics | Participants figure out what the researcher wants and give it | Reporting less anxiety to please the counselor |
Random assignment handles selection and, over time, regression. A control group handles history, maturation and testing, because both groups experience them equally. That is why the pretest-posttest control group design is considered so strong: most threats affect both groups the same way and cancel out.
Who is in the study? Sampling
| Method | How it works | Note |
|---|---|---|
| Simple random | Everyone in the population has an equal chance | The gold standard for external validity |
| Stratified random | Divide the population into subgroups (strata), then sample randomly from each in proportion | Guarantees small groups are represented |
| Cluster | Randomly pick whole groups (schools, clinics), then study everyone in them | Practical when you cannot list every individual |
| Systematic | Every k-th person from a list | Random start, then fixed interval |
| Convenience | Whoever is available | Most counseling research; weakest for generalizing |
| Purposive | Deliberately chosen because they have a characteristic of interest | Common in qualitative research |
| Snowball | Participants recruit other participants | Hidden or hard-to-reach populations |
Protecting participants
Every study involving people needs approval from an Institutional Review Board (IRB), informed consent from participants, the right to withdraw without penalty, protection of confidentiality, and debriefing if any deception was used. The Belmont Report (1979) set out the three principles: respect for persons, beneficence, and justice. The ACA Code covers research in Section G.
How the exam asks about this
Name the variable. "A researcher tests whether a new grief curriculum reduces depression scores." Curriculum is the IV, depression score is the DV.
Name the design. Look first for random assignment. Present: true experiment. Absent but a treatment was applied to existing groups: quasi-experiment. Nothing applied, just measured: correlational. One person measured over time with the treatment on and off: single-subject.
Spot the threat. "Half the control group dropped out before the posttest." Mortality. "Clients were chosen because they had the highest anxiety scores, and their scores fell." Statistical regression. "A hurricane struck the area during the study." History.
Can they say "caused"? Only with random assignment. Any option that draws a causal conclusion from a correlational or quasi-experimental study is wrong.
Without scrolling up, write everything you remember from this lesson. Then check yourself against the key terms. Recalling the whole lesson once, right after reading it, does more than reading it twice.
Ten more questions from the whole domain, with an explanation after each answer.
{{ doneNote }}