Deep dive ・ 8 min
How strong is the evidence? Trials, natural experiments
In a US job-training study, many estimates from observational data failed to match a randomised trial. The strengths and limits of each kind of study, and what to check.

The evidence behind this article
- Randomised trial1
- Research synthesis2
- Adjusted observational1
Includes studies built to test cause and effect (randomised trials or natural experiments).
4 sources・Countries studied: United States, Several countries
What to do with this
Check each article's kind of evidence: read trials and natural experiments as cause and effect, observational data as tendencies. Ask who was compared with whom.
Every article on this site shows the kinds of evidence it rests on: randomised trial, natural experiment, adjusted observational, observational or research synthesis. These labels describe how far a study can go. Using a well-known US job-training study as a guide, here are the strengths and limits of each kind and what to check.
Key points
- In a US job-training programme (National Supported Work), a trial that assigned people by lottery served as a benchmark. When standard observational methods were tried with comparison people drawn from survey data, many failed to reproduce the trial result (LaLonde 1986).
- A later study of the same data (Dehejia & Wahba 1999) got much closer to the trial result by comparing participants only with similar people, but depended on the assumption that everything driving participation was in the data.
- Answers from trials and natural experiments are also tied to a time and place, and a general picture emerges only as studies accumulate (Angrist & Pischke 2010). Research syntheses show averages, but in one review of 607 effect sizes, the average was positive while over a third went the other way.
How did a lottery-based trial test observational methods?
The strength of a randomised trial is that, because a lottery decides who takes part, the two groups differ on average only in whether they took part.
National Supported Work was a US employment and training programme run as a field experiment, with participants randomly assigned to training or not. LaLonde (1986) treated the trial’s result as the right answer, replaced the lottery control group with people drawn from survey data, and re-estimated the effect with the statistical methods common in evaluations at the time.
Many of those methods failed to reproduce the trial result. Angrist and Pischke describe how the estimates often differed quite a bit from the randomised result, and how LaLonde argued that comparing statistical models against one another was unlikely to point researchers in the right direction.
Trials have weaknesses too: they take time and money, and many questions cannot be settled by lottery.
How close can adjusted observational data get to the trial answer?
Dehejia and Wahba (1999) used the same data to try propensity score methods, which compare participants only with people whose characteristics before the programme were very similar.
Their estimates came much closer to the trial result than the methods LaLonde examined. But the authors spell out the conditions: participants and comparison people must overlap in their characteristics, and everything that drives participation must be in the data. The results were sensitive to that second assumption.
So adjustment can help, but whether its assumptions hold is hard to check in an ordinary study that has no trial result to compare against.
What do natural experiments and research syntheses add, and where do they struggle?
Natural experiments use situations in which a rule or event split people regardless of their circumstances. Angrist and Pischke cite a study of the Mariel boatlift, when Cuban migrants increased Miami’s labour force by about 7% in three months. They get close to cause and effect where a lottery is impossible, but must show that the split really was as good as random.
Whether from a trial or a natural experiment, an answer belongs to a particular time and place. The Tennessee STAR experiment cut class sizes from about 25 to about 15, but that need not tell us the effect of going from 40 to 30, and it might be specific to Tennessee. The authors’ answer is accumulation: studies from several countries and grade levels now give similar results, with a ten-student cut in class size raising test scores by about 0.2 to 0.3 standard deviations.
A research synthesis, or meta-analysis, pools many studies into an average and reduces reliance on any single one. But biases in the pooled studies carry into the average, and the spread matters too. A review of feedback interventions, pooling 607 effect sizes from 23,663 observations, found that feedback improved performance on average, yet over a third of the interventions made performance worse.
How should you read the “kind of evidence” labels on this site?
Randomised trials and natural experiments are built to test cause and effect, so articles describe them as “doing X increased Y”, still an average for the countries, people and period studied. Adjusted observational and observational data support only “people who do X tend to Y”. Research syntheses reduce reliance on any single study but depend on what they pool. Estimates and official guidelines are calculations based on assumptions, or expert guidance.
What this evidence supports is this: the same question can get different answers from different designs, and a lottery-based trial is a yardstick for the gap. The examples are from the US; Japanese studies making the same comparison are not covered here. Other articles cover correlation versus causation and failed replications.
A small step for today
- Before reading an article, look at its kind of evidence. Read trials and natural experiments as cause and effect, and observational data as tendencies.
- When you hear “research shows”, check four things: was there a lottery, who was the comparison group, which country and people, and how many studies.
- When you test a habit on yourself, log it with “Wrote one line on today’s progress.” Remember that one person’s log has no comparison group and is not a substitute for research.
FAQ
If a randomised trial found an effect, will it apply to me?
Not necessarily. A trial shows the average effect for one country, group and period. As the Tennessee class-size trial shows, a different setting may give a different effect, so check whether studies elsewhere agree.
Does this mean observational studies cannot be trusted?
No. In the US job-training data, comparing participants only with similar people came much closer to the trial result. But that relies on everything driving participation being in the data, and supports only “people who do X tend to Y”.
Is a meta-analysis always the most reliable evidence?
Pooling studies reduces reliance on any one of them, but an average can hide a wide spread: in the feedback review, over a third went the other way. Biases in the pooled studies carry into the average, so what kinds of studies were pooled also matters.
Sources
- [X-lalonde-1986] LaLonde (1986) Evaluating the Econometric Evaluations of Training Programs with Experimental Data. American Economic Review.
- [X-dehejia-wahba-1999] Dehejia & Wahba (1999) Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training Programs. Journal of the American Statistical Association.
- [X-angrist-pischke-2010] Angrist & Pischke (2010) The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics. Journal of Economic Perspectives.
- [E6] Kluger & DeNisi (1996) The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin.
Studies describe average tendencies. This article is not a promise that a habit will lift your income, and it is not medical or investment advice. About these articles ・ Search & sources
Updated:
← All articles