Deirdre McCloskey is very clearly a good economic historian, and a good rhetorician, occasional lapses into sexism be excused. We would all be saved much trouble if more economists would take her advice from “Economical Writing”, and saved further still if a 20 page limit were imposed on articles in our top journals: the 104 pages devoted to a single article on repeated games, thorough though it may be, seems a bit ridiculous when The Book of Genesis runs only 70 or so. The present book, on three “vices” of economists, is influential but in general should not be. This post explains why.
McCloskey’s first vice, which she has written about at length in a different book, concerns statistical significance. She gives an example of a 1980s AER which found a point estimate of a benefit:cost ratio of 4 for men from a policy, but stated that “the policy only affected women” because 4 was not big enough to be statistically significant at the 5 percent level given the sample size. And of course McCloskey is right: statements like these are completely wrong. But let’s break things down more finely. Error 1: Claiming an effect is zero when statistics merely tell us we can’t rule out a zero effect coming from pure luck. Error 2: Fetishizing “significant” results in an of themselves, particularly when reviewers implicitly make Error 1 when evaluation a paper. Error 3: Thinking that sample error is the only error creeping into an estimate, when clearly misspecification error, mistakes in the data, etc., are all always present. Error 4: Finding a “significant” result and calling it a day, without explicitly discussing how important the estimate is; David Cutler’s explicit loss functions in his impact of health outcome papers are a great example of avoiding this error. All four errors are definitely present in economics, probably worse in medicine and our fellow social sciences, and ought be excised. But saying all that, I don’t think simulation, or “moment matching”/calibration, or other methods that avoid discussions of sample error are any better; we should be skeptical of large point estimate effects that come with huge standard errors! This is a sign of massive variation in the data of whatever effect we’re trying to capture. The “significance” itself is not important, but the sample variation is. When McCloskey says that sample error only matters if we have an explicit loss function incorporating sample error, econometricians ought argue back that science and policy and knowledge do have implicit loss functions, and rightfully give weight to results produced with less sample error.
Her second vice is the “Samuelsonian” vice of blackboard theorizing. She is not anti-mathematical, but she does not see much value from huge portions of the economic academy proving that axioms A imply C, then showing A’ implies C’, then showing A” implies C”. I guess two responses should be posed here. First, consider purely empirical work. We are trying to learn, utterly atheoretically, whether minimum wage and employment are linked, in general. We don’t especially care about the link in New Jersey in 1988, or in Cambodia in 1963, or in Los Angeles in 1996. We want to learn something general about the link in order to guide future policy. If the New Jersey study finds a positive effect, the Cambodia study no effect, and the LA study a negative effect, would anyone argue that the studies were just “sandbox theorizing”, just using data A to prove C, then using data A’ to prove C’, then using A” to prove C”? Of course not. We would argue that finding a general link requires exploring the data space, replicating studies, trying to understand the differences in each individual circumstance. And such work would be useful! What theory does is examine the data space in exactly the same manner, using axioms to derive general “laws” of human behavior.
The second response involves those “laws”, because the obvious retort here is that, as Kahneman noted, it is easy to construct an experiment to “disprove” any social science law. My philosophic background is highly analytic, so when argument devolve in that way, my first thought is to examine the language game. Here, social scientists are using “law” in a different way from physicists: we know they are not true, but we think they capture important facets of general human behavior. Perhaps we ought call these “tendencies” rather than laws. And understanding these theoretical tendencies is actually critical to doing empirical work. I gave a presentation on the importance of theory to empirical health economics last year, using the following example. Consider a state that began hospital report cards, putting hospital data and evaluations online. Should we see any impact on health outcomes? If I find data from two years later showing no change in behavior by either hospitals or patients, what should I infer? You might infer that report cards are not useful, and stop the program. This would be mistaken. The theory of reputational games has a number of results that basically show there is a discontinuous equilibrium: no one cares about report cards unless a sufficiently large number of hospitals get patients who care. These multiple equilibria are actually quite delicate: they are only known because of a long detour through games, then repeated games, then reputational games with short and long-term players, all of which have been associated with various monitoring strategies. But we do not need to reprove these results for schools and for local governments and for other areas where reputation is important: the report is proven, and proven for all time.
One final note on the Samuelsonian vice: McCloskey wrote this book early in the era of agent-based modeling, and she is very positive about simulation. I think ABM – and I think practitioners of simulation would largely agree – is useful only when an underlying “math-department value” proof is impossible. I can give you a great example. A family member emailed me a link from the MIT Press blog to an article by some Italian physicists on the Peter Principle. The Peter Principle says that people are promoted to their level of maximum incompetence. Basically, if your skill in every job in an organization is uncorrelated, and you are promoted when you do well, then everyone will be promoted out of jobs they are good at, and will stop being promoted once they are terrible at their job. On the other hand, if your skill at each level of the organization is correlated, you want your best people in higher positions where they have more impact on the firm’s bottom line. The physicists’ article runs a simulation of a 6-level organization under the “no correlation” and “lots of correlation” assumptions, and the simulation unsurprisingly tells you exactly the results in the previous sentence. But this simulation is totally useless – someone utterly incompetent in statistics like myself was able to prove, mathematically, the same result in less than a half an hour. The full mathematical proof tells you exactly how much correlation, given the organization, you need between jobs to avoid the Peter Principle. It does all the extensions for you. And it is no more difficult – indeed, it is probably less difficult. I am glad that the simulation article would have zero chance of being published in even marginal economics articles. [I hate to tell you the denouement: the article was published in a very good physics journal, Physics A. Woe is us.]
McCloskey’s final vice, named for Jan Tinbergen, is attempts at social engineering. It’s very rare in being an unclear essay by a normally very good writer. At various times in the chapter, Tinbergen’s vice is large-scale models of the economy, attempts to beat the efficient market hypothesis, and programs which attempt to use social science results to guide society from on high. The second seems pretty uncontroversial, but not the first and the third. Klein style models of the economy, for all their faults, actually are useful for running counterfactual simulation, and absolutely are used internally at the Fed and at projection companies like Macro Advisers. As for guiding society from on high, in many cases the question is how to guide society, not whether to do so. The question is how to lay out the roads and transit network, not whether to do so. The question is who to give the property rights to, even if Coase will get us to efficiency. Surely none of this is objectionable to a classical liberal, and surely social scientists can say something about these questions without leading us on the Road to Serfdom.
Caveats aside, grab a copy of this short book from your library: the food for thought is worth it.
http://www.amazon.com/Vices-Economists-Virtues-Bourgeoisie/dp/9053562443/ (Amazon link; it’s a bit pricey even used, so just head to your library. Would that McCloskey put copies of out-of-print books on her website!)
A Fine Theorem 
Can you share your proof of the Peter Principle with the readers of your blog?