Leamer (1983) famously said that nobody takes anyone else’s data analysis seriously in economics. At the time, applied microeconometrics generally tried to uncover causality by simple regression with undefended instruments. By changing the regression specification or using a different instrument, results could often be wholesale flipped. Leamer’s suggested fix was sensitivity analysis, whereby results for the parameter of interest are given using a number of different specifications. In macro, things were more devastating for data analysis, since the Lucas Critique suggested that forward-looking optimizing agents meant that causal relations could not be uncovered from data alone. Nonetheless, data work is common and respected today, and sensitivity analysis is rarely used. What happened?
Angrist and Pischke claim that the credibility revolution was a function of better research design. In particular, randomized trials and natural experiments are widespread in economics today, and by construction they immediately give causal results. The authors claim that IO is the last field unconquered by natural experiments. They also attempt to refute the claim that random experiments means that economists are studying “trivial matters” such as the demand for snow boots. Since some areas have better data or useful natural experiments, the authors claim that external validity is gained by the accumulation of internally valid results across, say, many different products, or many different class sizes, or many different countries for development work. This strikes me as a question of philosophical inference; the accumulation of data certainly is “convincing”, but whether it can lead to knowledge of “truth” is, I think, an idea that most philosophers reject. At that level, theoretical arguments, being simply exercises in logic once assumptions are accepted, strike me as more useful, though given that there is a place for empirical work, there is no particular flaw with the “accumulation” argument that wouldn’t also strike down other methods of ascertaining global truths from nonglobal data.